System and method for teaching compositionality to convolutional neural networks

Abstract

A system for teaching compositionality to convolutional neural networks includes an unmasked convolutional neural network comprising a first set of convolutional neural network layers; a first masked convolutional neural network comprising a second set of convolutional neural network layers; the unmasked convolutional neural network and the first masked convolutional network sharing convolutional neural network weights; the system training the unmasked and first masked convolutional neural networks simultaneously based on an objective function that seeks to reduce both discriminative loss and compositional loss.

We claim: 
     
         1 . A system comprising a non-transitory computer readable medium having stored thereon software instructions that, when executed by a processing system, cause the processing system to train a first and second set of convolutional neural network layers by:
 determining a masked input, comprising masking a training input with a first mask;   with the masked input and using the second set of convolutional neural network layers, determining a first result;   with the training input and using the first set of convolutional neural network layers:
 determining intermediate outputs of a layer of the first set; and 
 based on the intermediate outputs, determining a second result; 
   determining compositional loss based on the intermediate outputs masked with a second mask;   determining discriminative loss based on the first and second results; and   training both the first and second sets of convolutional neural network layers based on both the discriminative loss and the compositional loss.   
     
     
         2 . The system of  claim 1 , wherein the first and second sets of convolutional neural network layers share convolutional neural network weights. 
     
     
         3 . The system of  claim 1 , wherein the first mask is associated with the second mask. 
     
     
         4 . The system of  claim 3 , wherein the first mask and the second mask suppress activation of identical regions. 
     
     
         5 . The system of  claim 1 , wherein the first mask suppresses a first region of a training input; wherein the second mask suppresses activation of at least a subset of the first region. 
     
     
         6 . The system of  claim 1 , wherein the first mask comprises an object mask and the second mask comprises a second object mask. 
     
     
         7 . The system of  claim 1 , wherein the first and second sets of convolutional neural network layers are trained based on an objective function that seeks to reduce both discriminative loss and compositional loss. 
     
     
         8 . The system of  claim 7 , herein objective function models discriminative loss using softmax-cross entropy for joint class prediction. 
     
     
         9 . The system of  claim 7 , wherein the objective function models discriminative loss using sigmoid-cross entropy for independent class prediction. 
     
     
         10 . The system of  claim 1 , wherein the first and second sets of convolutional neural network layers share all convolutional neural network weights. 
     
     
         11 . The system of  claim 1 , wherein training both the first and second sets of convolutional neural network layers comprises simultaneously updating weights of the first and second sets using a gradient descent algorithm. 
     
     
         12 . The system of  claim 1 , wherein determining the second result comprises determining corresponding intermediate outputs of a corresponding layer of the second set of convolutional neural network layers, wherein the corresponding layer of the second set corresponds to the first layer of the first set, wherein determining the compositional loss comprises:
 determining masked intermediate outputs with the intermediate outputs using the second mask; and   comparing the masked intermediate outputs with the corresponding intermediate outputs of the corresponding layer of the second set.   
     
     
         13 . The system of  claim 1 , wherein the compositional loss is further determined based on a second set of intermediate outputs for an other layer of the first set. 
     
     
         14 . A method for a convolutional neural network comprising:
 determining a masked input, comprising masking a training input with a first mask;   with the training input and using a first set of convolutional neural network layers, determining intermediate outputs of a layer of the first set of convolutional neural network layers;   determining compositional loss based on the intermediate outputs and a second mask;   determining discriminative loss from respective outputs of the first set of convolutional network layers and a second set of convolutional neural network layers based on the training input and masked input, respectively; and   updating convolutional neural network weights of both the first and second sets of convolutional neural network layers based on the compositional loss and the discriminative loss;   wherein:   the first and second sets of convolutional neural network layers share convolutional neural network weights.   
     
     
         15 . The system of  claim 14 , wherein the second mask suppresses activation of a subset of the first region. 
     
     
         16 . The system of  claim 14 , wherein the first mask comprises an object mask and the second mask comprises a second object mask. 
     
     
         17 . The system of  claim 16 , wherein the first mask and the second mask suppress activation of identical objects. 
     
     
         18 . The system of  claim 14 , wherein the first and second sets of convolutional neural network layers are trained based on an objective function that seeks to reduce both discriminative loss and compositional loss. 
     
     
         19 . The system of  claim 14 , wherein the first and second sets of convolutional neural network layers share all convolutional neural network weights. 
     
     
         20 . The system of  claim 14 , further comprising determining an additional set of intermediate outputs for the other layer of the first set of convolutional neural network layers, wherein determining the compositional loss comprises: masking the intermediate outputs and the additional intermediate outputs.