Generating synthetic training data including document images with key-value pairs

Abstract

Automated techniques are for generating a large volume of diverse training data that can be used for training machine learning models to extract KV pairs from document images. Given a single input document image and associated annotation data, a large number of diverse synthetic training datapoints are automatically generated by a synthetic data generation system, each datapoint including a synthetic document image and associated annotation data. The generated synthetic training datapoints can be used to train and improve the performance of ML models for extracting KV pairs from document images. In certain implementations, multiple synthetic datapoints are generated by varying the values associated with a key for a content item within the input document image.

What is claimed is: 
     
       1. A method comprising:
 obtaining, by a synthetic data generation system (SDGS), a result of performing optical character recognition (OCR) on an input document image comprising text comprising a plurality of content items having a plurality of values, respectively, the result comprising information indicative of the plurality of values and information identifying, for each of the plurality of values, a location within the input document image of a content item from the plurality of content items that corresponds to a value of the plurality of values, wherein the plurality of content items comprises a first content item and the plurality of values comprises a value corresponding to the first content item, 
 wherein the obtaining the result of the OCR comprises:
 dividing the text into text units by performing the OCR on the text, each of the text units corresponding to one of the plurality of content items and is enclosed by a bounding box, and 
 extracting the text units and location information of four corners of each bounding box as the location of each of the plurality of content items, the location information comprising x, y coordinates of the four corners; 
 
 receiving, by the SDGS, an annotation to the result, the annotation indicating that the value from the plurality of values is associated with a first key; 
 determining, by the SDGS, a plurality of synthetic values for the first key, the plurality of synthetic values comprising a first synthetic value different from the value and a second synthetic value different from the value and from the first synthetic value; 
 generating, by the SDGS, a plurality of synthetic document images including:
 a first synthetic document image comprising a first set of content items including the first content item and one or more second content items from the plurality of content items, wherein the first synthetic document image includes the first synthetic value for the first content item, and, for the one or more second content items, one or more second values from the plurality of values that correspond to the one or more second content items and were included in the input document image, and 
 a second synthetic document image comprising a second set of content items including the first content item and one or more third content items from the plurality of content items, wherein the second synthetic document image includes the second synthetic value for the first content item, and, for the one or more third content items, one or more third values from the plurality of values that correspond to the one or more third content items and were included in the input document image; and 
 
 generating, by the SDGS, a plurality of annotation data for the plurality of synthetic document images, the plurality of annotation data including:
 first annotation data for the first synthetic document image, the first annotation data comprising, for each content item in the first set of content items, information indicative of a corresponding value included in the first synthetic document image, and information identifying a corresponding location within the first synthetic document image of the content item, and 
 second annotation data for the second synthetic document image, the second annotation data comprising, for each content item in the second set of content items, information indicative of a corresponding value included in the second synthetic document image, and information identifying a corresponding location within the second synthetic document image of the content item, 
 
 wherein the method further comprises:
 generating a plurality of synthetic training datapoints, each of the plurality of synthetic training datapoints comprising a corresponding synthetic document image among the plurality of synthetic document images and associated annotation data among the plurality of annotation data; and 
 outputting the plurality of synthetic training datapoints to another computer system configured to train a machine learning (ML) model using the plurality of synthetic training datapoints and generate the trained ML model, wherein the trained ML model is configured to, based on a provided document image as an input, (1) identify a certain key in the provided document image and (2) identify a value in the provided document image that corresponds to the certain key. 
 
 
     
     
       2. The method of  claim 1 , wherein:
 the determining the plurality of synthetic values comprises determining the first synthetic value and the second synthetic value using a key-value (KV) content database that stores a plurality of historical values, 
 each of the plurality of historical values in the KV content database is associated with one of a plurality of historical keys, to form historical KV pairs, and 
 the first key is one of the plurality of historical keys. 
 
     
     
       3. The method of  claim 2 , further comprising:
 searching the KV content database to identify historical values corresponding to the first key among the plurality of historical values, 
 wherein the first synthetic value and the second synthetic value are the identified historical values. 
 
     
     
       4. The method of  claim 1 , wherein:
 the receiving the annotation to the result comprises receiving a plurality of annotations, the plurality of annotations indicating that values corresponding to some of the plurality of content items are associated with a plurality of particular keys, and 
 the method further comprises:
 prior to the determining the plurality of synthetic values for the first key, receiving, by the SDGS, a user input for specifying the first key as a key for which the plurality of synthetic values are to be determined and the plurality of synthetic document images are to be generated. 
 
 
     
     
       5. The method of  claim 1 , further comprising:
 receiving, by the SDGS, a user input for specifying a number of the plurality of synthetic training datapoints to be generated. 
 
     
     
       6. The method of  claim 1 , wherein the generating the plurality of synthetic document images comprises:
 inserting, into at least one from among the first synthetic document image and the second synthetic document image, a background image. 
 
     
     
       7. The method of  claim 6 , wherein the background image is a logo. 
     
     
       8. The method of  claim 1 , wherein the generating the plurality of synthetic document images comprises:
 changing, for at least one from among the first synthetic document image and the second synthetic document image, at least one from among a font size and a font style. 
 
     
     
       9. The method of  claim 1 , wherein:
 the input document image includes one from among a receipt image and an invoice image, and 
 the generating the plurality of synthetic document images comprises generating the plurality of synthetic document images corresponding to the one from among the receipt image and the invoice image. 
 
     
     
       10. The method of  claim 1 , wherein the obtaining the result of the OCR on the input document image further comprises:
 obtaining an OCR image including rows, each of the rows including one of the plurality of content items and location information corresponding to the one of the plurality of content items. 
 
     
     
       11. The method of  claim 10 , wherein the receiving the annotation comprises obtaining the OCR image to which the first key is added in correspondence to the first content item located in one of the rows. 
     
     
       12. The method of  claim 11 , further comprising:
 prior to the generating the plurality of synthetic document images, generating, by the SDGS, a template based on the OCR image to which the first key is added, the generating the template comprising masking the value corresponding to the first content item in the one of the rows, and generating the template comprising, in the one of the rows, the first key, an empty value field corresponding to the masked value, and location information corresponding to the first content item, 
 wherein the generating the first synthetic document image comprises:
 associating the first synthetic value with the empty value field, to generate a first synthetic template, based on which the first synthetic document image is generated, and 
 associating the second synthetic value with the empty value field, to generate a second synthetic template, based on which the second synthetic document image is generated. 
 
 
     
     
       13. The method of  claim 1 , wherein the generating the plurality of synthetic document images comprises generating the plurality of synthetic document images in parallel, partially in parallel, or successively. 
     
     
       14. The method of  claim 1 , wherein:
 the trained ML model is configured to identify the certain key from a predefined set of keys, and 
 the predefined set of keys is identified during training of the ML model using the plurality of synthetic training datapoints. 
 
     
     
       15. The method of  claim 1 , further comprising:
 receiving a user selection input through a user interface, the user selection input providing a selection of a set of keys, 
 wherein the certain key is a key included in the set of keys. 
 
     
     
       16. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more computer systems of a synthetic data generation system (SDGS), cause the SDGS to perform a method including:
 obtaining a result of performing optical character recognition (OCR) on an input document image comprising text comprising a plurality of content items having a plurality of values, respectively, the result comprising information indicative of the plurality of values and information identifying, for each of the plurality of values, a location within the input document image of a content item from the plurality of content items that corresponds to a value of the plurality of values, wherein the plurality of content items comprises a first content item and the plurality of values comprises a value corresponding to the first content item, 
 wherein the obtaining the result of the OCR includes:
 dividing the text into text units by performing the OCR on the text, each of the text units corresponding to one of the plurality of content items and is enclosed by a bounding box, and 
 extracting the text units and location information of four corners of each bounding box as the location of each of the plurality of content items, the location information comprising x, y coordinates of the four corners; 
 
 receiving an annotation to the result, the annotation indicating that the value from the plurality of values is associated with a first key; 
 determining a plurality of synthetic values for the first key, the plurality of synthetic values comprising a first synthetic value different from the value and a second synthetic value different from the value and from the first synthetic value; 
 generating a plurality of synthetic document images including:
 a first synthetic document image comprising a first set of content items including the first content item and one or more second content items from the plurality of content items, wherein the first synthetic document image includes the first synthetic value for the first content item, and, for the one or more second content items, one or more second values from the plurality of values that correspond to the one or more second content items and were included in the input document image, and 
 a second synthetic document image comprising a second set of content items including the first content item and one or more third content items from the plurality of content items, wherein the second synthetic document image includes the second synthetic value for the first content item, and, for the one or more third content items, one or more third values from the plurality of values that correspond to the one or more third content items and were included in the input document image; and 
 
 generating a plurality of annotation data for the plurality of synthetic document images, the plurality of annotation data including:
 first annotation data for the first synthetic document image, the first annotation data comprising, for each content item in the first set of content items, information indicative of a corresponding value included in the first synthetic document image, and information identifying a corresponding location within the first synthetic document image of the content item, and 
 second annotation data for the second synthetic document image, the second annotation data comprising, for each content item in the second set of content items, information indicative of a corresponding value included in the second synthetic document image, and information identifying a corresponding location within the second synthetic document image of the content item, 
 
 wherein the method further includes:
 generating a plurality of synthetic training datapoints, each of the plurality of synthetic training datapoints comprising a corresponding synthetic document image among the plurality of synthetic document images and associated annotation data among the plurality of annotation data; and 
 outputting the plurality of synthetic training datapoints to another computer system configured to train a machine learning (ML) model using the plurality of synthetic training datapoints and generate the trained ML model, wherein the trained ML model is configured to, based on a provided document image as an input, (1) identify a certain key in the provided document image and (2) identify a value in the provided document image that corresponds to the certain key. 
 
 
     
     
       17. The non-transitory computer-readable medium of  claim 16 , wherein:
 the determining the plurality of synthetic values includes determining the first synthetic value and the second synthetic value using a key-value (KV) content database that stores a plurality of historical values, 
 each of the plurality of historical values in the KV content database is associated with one of a plurality of historical keys, to form historical KV pairs, and 
 the first key is one of the plurality of historical keys. 
 
     
     
       18. The non-transitory computer-readable medium of  claim 17 , wherein:
 the method further includes searching the KV content database to identify historical values corresponding to the first key among the plurality of historical values, 
 wherein the first synthetic value and the second synthetic value are the identified historical values. 
 
     
     
       19. The non-transitory computer-readable medium of  claim 16 , wherein:
 the receiving the annotation to the result includes receiving a plurality of annotations, the plurality of annotations indicating that values corresponding to some of the plurality of content items are associated with a plurality of particular keys, and 
 the method further includes:
 prior to the determining the plurality of synthetic values for the first key, receiving a user input for specifying the first key as a key for which the plurality of synthetic values are to be determined and the plurality of synthetic document images are to be generated. 
 
 
     
     
       20. A system comprising:
 one or more computer systems configured to perform a method including: 
 obtaining a result of performing optical character recognition (OCR) on an input document image comprising text comprising a plurality of content items having a plurality of values, respectively, the result comprising information indicative of the plurality of values and information identifying, for each of the plurality of values, a location within the input document image of a content item from the plurality of content items that corresponds to a value of the plurality of values, wherein the plurality of content items comprises a first content item and the plurality of values comprises a value corresponding to the first content item, 
 wherein the obtaining the result of the OCR includes:
 dividing the text into text units by performing the OCR on the text, each of the text units corresponding to one of the plurality of content items and is enclosed by a bounding box, and 
 extracting the text units and location information of four corners of each bounding box as the location of each of the plurality of content items, the location information comprising x, y coordinates of the four corners; 
 
 receiving an annotation to the result, the annotation indicating that the value from the plurality of values is associated with a first key; 
 determining a plurality of synthetic values for the first key, the plurality of synthetic values comprising a first synthetic value different from the value and a second synthetic value different from the value and from the first synthetic value; 
 generating a plurality of synthetic document images including:
 a first synthetic document image comprising a first set of content items including the first content item and one or more second content items from the plurality of content items, wherein the first synthetic document image includes the first synthetic value for the first content item, and, for the one or more second content items, one or more second values from the plurality of values that correspond to the one or more second content items and were included in the input document image, and 
 a second synthetic document image comprising a second set of content items including the first content item and one or more third content items from the plurality of content items, wherein the second synthetic document image includes the second synthetic value for the first content item, and, for the one or more third content items, one or more third values from the plurality of values that correspond to the one or more third content items and were included in the input document image; and 
 
 generating a plurality of annotation data for the plurality of synthetic document images, the plurality of annotation data including:
 first annotation data for the first synthetic document image, the first annotation data comprising, for each content item in the first set of content items, information indicative of a corresponding value included in the first synthetic document image, and information identifying a corresponding location within the first synthetic document image of the content item, and 
 second annotation data for the second synthetic document image, the second annotation data comprising, for each content item in the second set of content items, information indicative of a corresponding value included in the second synthetic document image, and information identifying a corresponding location within the second synthetic document image of the content item, 
 
 wherein the method further includes:
 generating a plurality of synthetic training datapoints, each of the plurality of synthetic training datapoints comprising a corresponding synthetic document image among the plurality of synthetic document images and associated annotation data among the plurality of annotation data; and 
 outputting the plurality of synthetic training datapoints to another computer system configured to train a machine learning (ML) model using the plurality of synthetic training datapoints and generate the trained ML model, wherein the trained ML model is configured to, based on a provided document image as an input, (1) identify a certain key in the provided document image and (2) identify a value in the provided document image that corresponds to the certain key.