Neural network based scene text recognition

Abstract

A system uses a neural network based model to perform scene text recognition. The system achieves high accuracy of prediction of text from scenes based on a neural network architecture that uses double attention mechanism. The neural network based model includes a convolutional neural network component that outputs a set of visual features and an attention extractor neural network component that determines attention scores based on the visual features. The visual features and the attention scores are combined to generate mixed features that are provided as input to a character recognizer component that determines a second attention score and recognizes the characters based on the second attention score. The system trains the neural network based model by adjusting the neural network parameters to minimize a multi-class gradient harmonizing mechanism (GHM) loss. The multi-class GHM loss varies based on a level of difficulty of the sample.

We claim: 
     
       1. A computer implemented method for performing scene text recognition using a neural network based model, the method comprising:
 receiving a request to recognize text in an input image, the input image comprising a scene with embedded text; 
 providing the input image to a convolutional neural network component to generate visual features and multi-scale visual features based on the input image, the visual features represented as a feature vector of lower dimension than the input image; 
 providing the visual features output by the convolutional neural network component to an attention extractor neural network component, wherein the attention extractor neural network component:
 summarizes the multi-scale visual features into summarized visual features; 
 generates attention features based on the summarized visual features of the input image, each attention feature corresponding to an attention map for a visual feature, and 
 adds attention maps with corresponding attention features to generate added attention features; 
 
 combining the visual features and the added attention features to obtain mixed features; 
 providing the mixed features as input to a character recognizer neural network component, wherein the character recognizer neural network component generates an attention score based on hidden features of the character recognizer neural network, the character recognizer neural network component outputting a sequence of characters representing predicted text recognized from the input image; and 
 providing the recognized text from the input image for display. 
 
     
     
       2. The method of  claim 1 , further comprising:
 receiving a training dataset for training the neural network based model, the training dataset comprising a plurality of samples; 
 for each character in a set of characters, determining a measure of difficulty of classifying the character based on a margin between a likelihood of correct prediction and a likelihood of incorrect prediction for the character based on the neural network based model; and 
 training the neural network based model using the training dataset, the training adjusting parameters of the neural network based model to minimize a loss value for samples of the training dataset, wherein a loss value for a sample is weighted according to the measure of difficulty assigned to the characters of the sample. 
 
     
     
       3. The method of  claim 1 , further comprising:
 transforming the input image by a text image rectifier to modify at least a portion of the input image comprising text, wherein the transformed image is provided as input to the convolutional neural network component. 
 
     
     
       4. The method of  claim 1 , wherein the convolutional neural network component comprises a residual neural network. 
     
     
       5. The method of  claim 1 , wherein the character recognizer neural network component comprises a binary long short-term memory (BLSTM) neural network that receives as input the mixed features and generates a set of spatial features. 
     
     
       6. The method of  claim 5 , wherein the set of spatial features is provided as input to a gated recurrent unit that predicts the character sequence. 
     
     
       7. The method of  claim 1 , wherein combining the visual features and the attention features to obtain a set of mixed features comprises determining a product of corresponding elements of the visual features and attention features and aggregating the products. 
     
     
       8. A computer implemented method for performing scene text recognition, the method comprising:
 receiving a training dataset for training a machine learning based model, the training dataset comprising a plurality of samples, each sample representing an image comprising a scene with embedded text based on a set of characters, the machine learning based model configured to receive an input image and predict a sequence of characters in the input image, wherein the machine learning based model comprising:
 a convolutional neural network component configured to receive the image to generate visual features and multi-scale visual features, the visual features represented as a feature vector of lower dimension than the image, 
 an attention extractor neural network component configured to:
 receive the multi-scale visual features output by the convolutional neural network, 
 summarize the multi-scale visual features into summarized visual features, 
 generate attention features based on the summarized visual features, each attention feature corresponding to an attention map for a visual feature, and 
 add attention maps with corresponding attention features to generate added attention features, and 
 
 a character recognizer neural network component configured to receive mixed features generated by combining the visual features and the added attention features to output a sequence of characters representing predicted text recognized from the image; 
 
 for each character in the set of characters, determining a measure of difficulty of classifying the character; 
 training the machine learning based model using the training dataset, the training adjusting parameters of the machine learning based model to minimize a loss value for samples of the training dataset, wherein a loss value for a sample is weighted according to the measure of difficulty assigned to the characters of the sample; and 
 predicting text in a new input image using the trained machine learning based model. 
 
     
     
       9. The method of  claim 8 , wherein the measure of difficulty of classifying the character is determined based on a margin between a likelihood of correct prediction and a likelihood of incorrect prediction for the character based on the machine learning based model. 
     
     
       10. The method of  claim 8 , wherein the measure of difficulty of a sample comprising a plurality of characters is determined as a sum of the measure of difficulty of each character from the plurality of characters. 
     
     
       11. The method of  claim 8 , wherein the machine learning based model is a neural network. 
     
     
       12. The method of  claim 11 , wherein the neural network comprises a convolutional neural network component to generate visual features based on the input image, the visual features represented as a feature vector of lower dimension than the input image. 
     
     
       13. The method of  claim 11 , the method further comprising:
 providing the visual features output by the convolutional neural network component to an attention extractor neural network component, wherein the attention extractor neural network component outputs attention features based on the visual features of the input image, each attention feature representing an attention map for a visual feature. 
 
     
     
       14. The method of  claim 13 , the method further comprising:
 combining the visual features and the attention features to obtain a set of mixed features and providing the mixed features as input to a character recognizer neural network component, wherein the character recognizer neural network component generates an attention score based on hidden features of the character recognizer neural network, the character recognizer neural network component outputting a sequence of characters representing predicted text recognized from the input image. 
 
     
     
       15. A computer system comprising:
 one or more computer processors; and 
 a non-transitory computer readable storage medium storing instructions that when executed by the one or more processors causes the one or more processors to perform operations comprising:
 receiving a request to recognize text in an input image, the input image comprising a scene with embedded text; 
 providing the input image to a convolutional neural network component to generate visual features and multi-scale visual features based on the input image, the visual features represented as a feature vector of lower dimension than the input image; 
 providing the multi-scale visual features output by the convolutional neural network component to an attention extractor neural network component, wherein the attention extractor neural network component:
 summarizes the multi-scale visual features into summarized visual features: 
 generates attention features based on the summarized visual features of the input image, each attention feature corresponding to an attention map for a visual feature, and 
 adds attention maps to corresponding attention features to generate added attention features; 
 
 combining the visual features and the added attention features to obtain mixed features; 
 providing the mixed features as input to a character recognizer neural network component, wherein the character recognizer neural network component generates an attention score based on hidden features of the character recognizer neural network, the character recognizer neural network component outputting a sequence of characters representing predicted text recognized from the input image; and 
 providing the recognized text from the input image for display. 
 
 
     
     
       16. The computer system of  claim 15 , wherein the instructions further cause the one or more processors to perform operations comprising:
 receiving a training dataset for training a neural network based model, the training dataset comprising a plurality of samples; 
 for each character in a set of characters, determining a measure of difficulty of classifying the character based on a margin between a likelihood of correct prediction and a likelihood of incorrect prediction for the character based on the neural network based model; and 
 training the neural network based model using the training dataset, the training adjusting parameters of the neural network based model to minimize a loss value for samples of the training dataset, wherein a loss value for a sample is weighted according to the measure of difficulty assigned to the characters of the sample. 
 
     
     
       17. The computer system of  claim 15 , wherein the instructions further cause the one or more processors to perform operations comprising:
 transforming the input image by a text image rectifier to modify at least a portion of the input image comprising text, wherein the transformed image is provided as input to the convolutional neural network component. 
 
     
     
       18. The computer system of  claim 15 , wherein the character recognizer neural network component comprises a binary long short-term memory (BLSTM) neural network that receives as input the mixed features and generates a set of spatial features. 
     
     
       19. The computer system of  claim 15 , wherein combining the visual features and the attention features to obtain a set of mixed features comprises determining a product of corresponding elements of the visual features and attention features and aggregating the products.