Content tagging

Abstract

Systems, methods, devices, media, and computer readable instructions are described for local image tagging in a resource constrained environment. One embodiment involves processing image data using a deep convolutional neural network (DCNN) comprising at least a first subgraph and a second subgraph, the first subgraph comprising at least a first layer and a second layer, processing, the image data using at least the first layer of the first subgraph to generate first intermediate output data; processing, by the mobile device, the first intermediate output data using at least the second layer of the first subgraph to generate first subgraph output data, and in response to a determination that each layer reliant on the first intermediate data have completed processing, deleting the first intermediate data from the mobile device. Additional embodiments involve convolving entire pixel resolutions of the image data against kernels in different layers if the DCNN.

What is claimed is: 
     
         1 . A system comprising:
 one or more processors; and   one or more memories storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:   accessing image data;   decompressing a first plurality of floating point weights;   processing, using the first plurality of floating point weights, the image data using a first layer of a deep convolutional neural network (DCNN) to generate first intermediate data;   decompressing a second plurality of floating point weights;   processing, using the second plurality of floating point weights, the first intermediate data using a second layer of the DCNN to generate first output data; and   causing the first intermediate data to be deleted from the system.   
     
     
         2 . The system of  claim 1 , wherein the operations further comprise:
 receiving a plurality of weights for the DCNN, wherein the plurality of weights are floating point weights compressed to weight indexes, wherein the decompressing the first plurality of floating point weights further comprises:   decompressing the first plurality of floating point weights from the plurality of weights indexes.   
     
     
         3 . The system of  claim 2 , wherein each weight index of the plurality of weight indexes indicates a floating point weight of the plurality of floating point weights. 
     
     
         4 . The system of  claim 1 , wherein the system is a mobile device. 
     
     
         5 . The system of  claim 1 , wherein the operation further comprise:
 capturing, using an image sensor of the system, the image data;   processing the image data as captured by the image sensor to generate a file comprising the image data at a first pixel resolution associated with a pixel height and a pixel width; and   storing the file in a second memory of the system.   
     
     
         6 . The system of  claim 5 , wherein the first layer comprises a convolutional layer, and wherein
 processing, using the first plurality of floating point weights, the image data comprises: convolving a first kernel with the image data, wherein the first kernel comprises a kernel pixel height less than the pixel height and a kernel pixel width less than the pixel width.   
     
     
         7 . The system of  claim 6 , wherein the first intermediate data comprises a plurality of matrixes, each matrix of the plurality of matrixes generated by convolving an associated kernel of a plurality of kernels of the first layer with the image data, wherein the plurality of kernels comprises the first kernel. 
     
     
         8 . The system of  claim 1 , wherein the operations further comprise:
 generating a plurality of output values from the first output data, each output value of the plurality of output values associated with a corresponding tag of a plurality of tags;   comparing the plurality of output values with a plurality of thresholds, the plurality of thresholds associated with the plurality of tags; and   assigning tags of the plurality of tags if corresponding output values of the plurality of output values transgress corresponding thresholds of the plurality of threshold.   
     
     
         9 . The system of  claim 8 , wherein the operations further comprise:
 accessing a plurality of metadata associated with the image data;   processing the assigned tags and the plurality of metadata using a natural language processor to generate extended visual search tags for the image data.   
     
     
         10 . The system of  claim 9  further comprising storing the image data and the extended visual search tags, in a memory of the system. 
     
     
         11 . The system of  claim 10  further comprising:
 receiving, via an input device, a search term; and 
 generating search results by comparing the search term with the extended visual search tags in the memory of the system. 
 
     
     
         12 . The system of  claim 1 , wherein a first subgraph comprises the first layer of the DCNN and the second layer of the DCNN, and wherein the operations further comprise:
 processing the first output data using a first layer of a second subgraph to generate second intermediate data;   processing the second intermediate data using a second layer of the second subgraph to generate second subgraph output data; and   in response to determining that processing of each layer associated with the second intermediate data is completed, causing the second intermediate data to be deleted from the system.   
     
     
         13 . The system of  claim 12 , wherein the operations further comprise:
 processing the second subgraph output data using a fully connected layer converted to a convolutional layer to generate a dense prediction score map.   
     
     
         14 . The system of  claim 13 , wherein the fully connected layer generates the dense prediction score map using an associated output from each convolution layer. 
     
     
         15 . The system of  claim 14 , wherein the operations further comprise:
 subsampling the dense prediction score map using a max-pooling operating to generate a plurality of output recognition scores, each output recognition score associated with one or more tags.   
     
     
         16 . The system of  claim 1 , wherein the first plurality of weights are 16 bit or 32 bit weights. 
     
     
         17 . A non-transitory storage medium comprising instructions that, when executed by one or more processors of a system, cause the system to perform operations, the operations comprising:
 accessing image data;   decompressing a first plurality of floating point weights;   processing, using the first plurality of floating point weights, the image data using a first layer of a deep convolutional neural network (DCNN) to generate first intermediate data;   decompressing a second plurality of floating point weights;   processing, using the second plurality of floating point weights, the first intermediate data using a second layer of the DCNN to generate first output data; and   causing the first intermediate data to be deleted from the system.   
     
     
         18 . The non-transitory storage medium of  claim 17 , wherein the operations further comprise:
 receiving a plurality of weights for the DCNN, wherein the plurality of weights are floating point weights compressed to weight indexes, wherein the decompressing the first plurality of floating point weights further comprises:   decompressing the first plurality of floating point weights from the plurality of weights indexes.   
     
     
         19 . A method perform by a system, the method comprising:
 accessing image data;   decompressing a first plurality of floating point weights;   processing, using the first plurality of floating point weights, the image data using a first layer of a deep convolutional neural network (DCNN) to generate first intermediate data;   decompressing a second plurality of floating point weights;   processing, using the second plurality of floating point weights, the first intermediate data using a second layer of the DCNN to generate first output data; and   causing the first intermediate data to be deleted from the system.   
     
     
         20 . The method of  claim 19  further comprising:
 receiving a plurality of weights for the DCNN, wherein the plurality of weights are floating point weights compressed to weight indexes, wherein the decompressing the first plurality of floating point weights further comprises: 
 decompressing the first plurality of floating point weights from the plurality of weights indexes.