Neural network sparsification device and method, and related product

Abstract

This disclosure relates to a device, a board card, a method and a readable storage medium for performing sparse training on a neural network model, wherein a processing device of the present disclosure is included in an integrated circuit device, and the integrated circuit device comprises a universal interconnection interface and a computation device. The computation device interacts with the processing device to jointly complete computing operations specified by the user. The integrated circuit device further comprises a storage device, and the storage device is connected to the computation device and the processing device, respectively, for data storage of the computation device and the processing device.

1 .- 30 . (canceled) 
     
     
         31 . A method of performing sparse training on a neural network model, comprising:
 in a mask adjustment stage, repeating following steps in a plurality of epochs:   masking, in forward propagation, mask adjustment parameters based on a mask tensor to compute a value of a loss function;   computing, in back propagation, partial derivatives of the loss function with respect to the mask adjustment parameters;   updating the mask adjustment parameters based on the partial derivatives; and   updating the mask tensor based on the updated mask adjustment parameters,
 wherein the updated mask adjustment parameters are shielded by using the updated mask tensor to control a processing area of a feature map input to the neural network model. 
   
     
     
         32 . The method of  claim 31 , further comprising:
 in a mask-free stage, repeating following steps in a plurality of epochs:   computing, in forward propagation, the value of the loss function based on mask-free parameters;   computing, in back propagation, partial derivatives of the loss function with respect to the mask-free parameters; and   updating the mask-free parameters based on the partial derivatives;   wherein the updated mask-free parameters are taken as initial values of the mask adjustment parameters.   
     
     
         33 . The method of  claim 32 , further comprising:
 randomly generating initial values of the mask tensor and the mask-free parameters.   
     
     
         34 . The method of  claim 31 , further comprising:
 determining the initial value of the mask tensor based on the initial values of the mask adjustment parameters.   
     
     
         35 . The method of  claim 34 , wherein when the mask tensor is a one-dimensional tensor, the determining the initial value of the mask tensor includes:
 selecting, from every m data elements of a specified dimension of the initial values of the mask adjustment parameters, n data elements with larger absolute values as valid data elements, wherein m>n; and   generating the initial value of the mask tensor based on positions of the n valid data elements in the m data elements.   
     
     
         36 . The method of  claim 35 , wherein the specified dimension is an input channel dimension. 
     
     
         37 . The method of  claim 34 , wherein when the mask tensor is a two-dimensional tensor, the determining the initial value of the mask tensor includes:
 presetting a specific count of two-dimensional mask tensors, wherein each dimension of the two-dimensional mask tensors includes m elements, of which n elements are 1 and m-n elements are 0, where m>n;   masking two specified dimensions of the initial values of the mask adjustment parameters of the neural network layer respectively based on each preset two-dimensional mask tensor to obtain masked parameter tensors;   performing product sum computation on training data of the neural network layer based on each masked parameter tensor to obtain parameter evaluation values; and   selecting a two-dimensional mask tensor that yields the largest of all the parameter evaluation values as the initial value of the mask tensor.   
     
     
         38 . The method of  claim 37 , wherein the two specified dimensions are an input channel dimension and an output channel dimension. 
     
     
         39 . The method of  claim 31 , wherein in the mask adjustment stage, the mask adjustment parameters are updated based on the partial derivatives in each iteration. 
     
     
         40 . The method of  claim 31 , wherein in the mask adjustment stage, when the mask tensor is a one-dimensional tensor, the updating the mask tensor includes:
 after a specific count of epochs have been performed, dividing the updated mask adjustment parameters into a plurality of intervals in units of a specific parameter count m;   sorting the mask adjustment parameters in each interval according to absolute values of the mask adjustment parameters from large to small;   setting, in the mask tensor, elements at positions that correspond to first n mask adjustment parameters with larger absolute values in each interval, to 1; and   setting, in the mask tensor, elements at positions that correspond to m-n mask adjustment parameters with smaller absolute values in each interval, to 0.   
     
     
         41 .- 44 . (canceled) 
     
     
         45 . A non-transitory computer-readable storage medium, having stored thereon computer program code for performing sparse training on a neural network model, which, when executed by a processing device, performs the method of  claim 31 . 
     
     
         46 . An integrated circuit device for performing sparse training on a neural network model, comprising:
 a processing device, comprising a control module, a computation module, and an updating module,   wherein when the control module sets that a mask adjustment stage is entered, the computation module repeats following operations in a plurality of epochs: masking, in forward propagation, mask adjustment parameters based on a mask tensor to compute a value of a loss function; and computing, in back propagation, partial derivatives of the loss function with respect to the mask adjustment parameters; and the updating module is configured to update the mask adjustment parameters based on the partial derivatives and update the mask tensor based on the updated mask adjustment parameters; and   a computation device configured to shield the updated mask adjustment parameters by using the updated mask tensor to control a processing area of a feature map input to the neural network model.   
     
     
         47 . The integrated circuit device of  claim 46 , wherein when the control module sets that a mask-free stage is entered, the computation module repeats following operations in a plurality of epochs: computing, in forward propagation, the value of the loss function based on mask-free parameters; and computing, in back propagation, partial derivatives of the loss function with respect to the mask-free parameters; and the updating module updates the mask-free parameters based on the partial derivatives, and takes the updated mask-free parameters as initial values of the mask adjustment parameters. 
     
     
         48 . The integrated circuit device of  claim 47 , wherein the processing device further includes a random generation module configured to randomly generate initial values of the mask tensor and the mask-free parameters. 
     
     
         49 . The integrated circuit device of  claim 46 , wherein the processing device further includes a mask tensor determination module configured to determine the initial value of the mask tensor based on the initial values of the mask adjustment parameters. 
     
     
         50 . The integrated circuit device of  claim 49 , wherein when the mask tensor is a one-dimensional tensor, the mask tensor determination module is configured to:
 select, from every m data elements of a specified dimension of the initial values of the mask adjustment parameters, n data elements with larger absolute values as valid data elements, wherein m>n; and   generate the initial value of the mask tensor based on positions of the n valid data elements in the m data elements.   
     
     
         51 . The integrated circuit device of  claim 50 , wherein the specified dimension is an input channel dimension. 
     
     
         52 . The integrated circuit device of  claim 49 , wherein when the mask tensor is a two-dimensional tensor, the mask tensor determination module is configured to:
 preset a specific count of two-dimensional mask tensors, wherein each dimension of the two-dimensional mask tensors includes m elements, of which n elements are 1 and m-n elements are 0, where m>n;   mask two specified dimensions of the initial values of the mask adjustment parameters of the neural network layer respectively based on each preset two-dimensional mask tensor to obtain masked parameter tensors;   perform product sum computation on training data of the neural network layer based on each masked parameter tensor to obtain parameter evaluation values; and   select a two-dimensional mask tensor that yields the largest of all the parameter evaluation values as the initial value of the mask tensor.   
     
     
         53 . The integrated circuit device of  claim 52 , wherein the two specified dimensions are an input channel dimension and an output channel dimension. 
     
     
         54 . The integrated circuit device of  claim 46 , wherein in the mask adjustment stage, the updating module updates the mask adjustment parameters based on the partial derivatives in each iteration. 
     
     
         55 .- 60 . (canceled)