Channel Gating For Conditional Computation

Abstract

A computing device may be equipped with a generalized framework for accomplishing conditional computation or gating in a neural network. The computing device may receive input in a neural network layer that includes two or more filters. The computing device may intelligently determine whether the two or more filters are relevant to the received input. The computing device may deactivate filters that are determined not to be relevant to the received input (or activate filters that are determined to be relevant to the received input), and apply the received input to active filters in the layer to generate an activation.

What is claimed is: 
     
         1 . A method for a neural network, comprising:
 receiving, by a processor in a computing device, input in a layer in the neural network, the layer including two or more filters;   determining whether the two or more filters are relevant to the received input;   deactivating filters that are determined not to be relevant to the received input or activating filters that are determined to be relevant to the received input; and   applying the received input to active filters in the layer to generate an activation.   
     
     
         2 . The method of  claim 1 , wherein each of the two or more filters in the layer are associated with a respective one of two or more gating functionality components. 
     
     
         3 . The method of  claim 2 , further comprising enforcing conditionality on the gating functionality components by back propagating a loss function to approximate a discrete decision of at least one of the gating functionality components with a continuous representation. 
     
     
         4 . The method of  claim 3 , wherein back propagating the loss function comprises performing Batch-wise conditional regularization operations to match batch-wise statistics of one or more of the gating functionality components to a prior distribution. 
     
     
         5 . The method of  claim 2 , wherein determining whether the two or more filters are relevant to the received input comprises:
 global average pooling the received input to generate a global average pooling result; and   applying the global average pooling result to the gating functionality components of each filter to generate a binary value for each filter.   
     
     
         6 . The method of  claim 5 , wherein deactivating filters that are determined not to be relevant to the received input or activating filters that are determined to be relevant to the received input comprise:
 identifying filters that may be ignored without impacting accuracy of the activation.   
     
     
         7 . The method of  claim 1 , wherein:
 receiving the input in the layer of the neural network comprises receiving the input in a convolution layer of a residual neural network (ResNet); and   deactivating filters that are determined not to be relevant to the received input or activating filters that are determined to be relevant to the received input comprises identifying filters associated with the convolution layer of the ResNet that may be ignored without impacting accuracy of the activation.   
     
     
         8 . The method of  claim 7 , wherein receiving the input in the layer of the neural network comprises receiving a set of three-dimensional input feature maps that form a channel of input feature maps in a layer that includes two or more three-dimensional filters. 
     
     
         9 . The method of  claim 8 , wherein applying the received input to active filters in the layer to generate an activation comprises:
 convolving the channel of input feature maps with one or more of the two or more three-dimensional filters to generate results; and   summing the generated results to generate the activation of the convolution layer as a channel of output feature maps.   
     
     
         10 . A computing device, comprising:
 a processor configured with processor-executable instructions to:
 receive input in a layer in a neural network, the layer including two or more filters; 
 determine whether the two or more filters are relevant to the received input; 
 deactivate filters that are determined not to be relevant to the received input or activate filters that are determined to be relevant to the received input; and 
 apply the received input to active filters in the layer to generate an activation. 
   
     
     
         11 . The computing device of  claim 10 , wherein the processor is further configured with processor-executable instructions to receive the input in a layer that includes two or more filters that are each associated with a respective one of two or more gating functionality components. 
     
     
         12 . The computing device of  claim 11 , wherein the processor is further configured with processor-executable instructions to enforce conditionality on the gating functionality components by back propagating a loss function to approximate a discrete decision of at least one of the gating functionality components with a continuous representation. 
     
     
         13 . The computing device of  claim 12 , wherein the processor is further configured with processor-executable instructions to back propagate the loss function comprises performing Batch-wise conditional regularization operations to match batch-wise statistics of one or more of the gating functionality components to a prior distribution. 
     
     
         14 . The computing device of  claim 11 , wherein the processor is further configured with processor-executable instructions to determine whether the two or more filters are relevant to the received input by:
 global average pooling the received input to generate a global average pooling result; and   applying the global average pooling result to each filter's associated gating functionality component to generate a binary value for each filter.   
     
     
         15 . The computing device of  claim 14 , wherein the processor is further configured with processor-executable instructions to deactivate the filters that are determined not to be relevant to the received input or activate the filters that are determined to be relevant to the received input by:
 identifying filters that may be ignored without impacting accuracy of the activation.   
     
     
         16 . The computing device of  claim 10 , wherein the processor is further configured with processor-executable instructions to:
 receive the input in the layer of the neural network by receiving the input in a convolution layer of a residual neural network (ResNet); and   deactivate filters that are determined not to be relevant to the received input or activate filters that are determined to be relevant to the received input by identifying filters associated with the convolution layer of the ResNet that may be ignored without impacting accuracy of the activation.   
     
     
         17 . A non-transitory processor-readable storage medium having stored thereon processor-executable instructions to cause a processor in a computing device executing a neural network to perform operations comprising:
 receiving an input in a layer in the neural network, the layer including two or more filters;   determining whether the two or more filters are relevant to the received input;   deactivating filters that are determined not to be relevant to the received input or activating filters that are determined to be relevant to the received input; and   applying the received input to active filters in the layer to generate an activation.   
     
     
         18 . The non-transitory processor-readable storage medium of  claim 17 , wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that receiving the input in the layer in the neural network comprises receiving the input in a layer that includes two or more filters that are each associated with a respective one of two or more gating functionality components. 
     
     
         19 . The non-transitory processor-readable storage medium of  claim 18 , wherein the stored processor-executable instructions are configured to cause the processor to perform operations further comprising enforcing conditionality on the gating functionality components by back propagating a loss function to approximate a discrete decision of at least one of the gating functionality components with a continuous representation. 
     
     
         20 . The non-transitory processor-readable storage medium of  claim 19 , wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that back propagating the loss function comprises performing Batch-wise conditional regularization operations to match batch-wise statistics of one or more of the gating functionality components to a prior distribution. 
     
     
         21 . The non-transitory processor-readable storage medium of  claim 18 , wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that determining whether the two or more filters are relevant to the received input comprises:
 global average pooling the received input to generate a global average pooling result; and   applying the global average pooling result to each filter's associated gating functionality component to generate a binary value for each filter.   
     
     
         22 . The non-transitory processor-readable storage medium of  claim 21 , wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that deactivating filters that are determined not to be relevant to the received input or activating filters that are determined to be relevant to the received input comprises:
 identifying filters that may be ignored without impacting accuracy of the activation.   
     
     
         23 . The non-transitory processor-readable storage medium of  claim 17 , wherein the stored processor-executable instructions are configured to cause the processor to perform operations such that:
 receiving the input in the layer of the neural network comprises receiving the input in a convolution layer of a residual neural network (ResNet); and   deactivating filters that are determined not to be relevant to the received input or activating filters that are determined to be relevant to the received input comprises identifying filters associated with the convolution layer of the ResNet that may be ignored without impacting accuracy of the activation.   
     
     
         24 . A computing device, comprising:
 means for receiving an input in a layer in a neural network, the layer including two or more filters;   means for determining whether the two or more filters are relevant to the received input;   means for deactivating filters that are determined not to be relevant to the received input or activating filters that are determined to be relevant to the received input; and   means for applying the received input to active filters in the layer to generate an activation.   
     
     
         25 . The computing device of  claim 24 , wherein means for receiving the input in a layer in the neural network comprises means for receiving the input in a layer that includes two or more filters that are each associated with a respective one of two or more gating functionality components. 
     
     
         26 . The computing device of  claim 25 , further comprising means for enforcing conditionality on the gating functionality components by back propagating a loss function to approximate a discrete decision of at least one of the gating functionality components with a continuous representation. 
     
     
         27 . The computing device of  claim 26 , wherein means for back propagating the loss function comprises means for performing Batch-wise conditional regularization operations to match batch-wise statistics of one or more of the gating functionality components to a prior distribution. 
     
     
         28 . The computing device of  claim 25 , wherein means for determining whether the two or more filters are relevant to the received input comprises:
 means for global average pooling the received input to generate a global average pooling result; and   means for applying the global average pooling result to each filter's associated gating functionality component to generate a binary value for each filter.   
     
     
         29 . The computing device of  claim 28 , wherein means for deactivating filters that are determined not to be relevant to the received input or activating filters that are determined to be relevant to the received input comprises:
 means for identifying filters that may be ignored without impacting accuracy of the activation.   
     
     
         30 . The computing device of  claim 24 , wherein:
 means for receiving the input in the layer of the neural network comprises means for receiving the input in a convolution layer of a residual neural network (ResNet); and   means for deactivating filters that are determined not to be relevant to the received input or activating filters that are determined to be relevant to the received input comprises means for identifying filters associated with the convolution layer of the ResNet that may be ignored without impacting accuracy of the activation.