Enhancing electronic form data based on hierarchical context information

Abstract

A form management server computer (“server”) and processing methods are disclosed. In some embodiments, the server is programmed to cause presentation of a result of identifying form elements from an image of a form and allow correction of the result or specification of actions to be performed on the form elements. The server is programmed to then receive a new image of a new form, identify form elements in the new form from the new image, and determine actions to be performed on the form elements in the new form based on the specification of actions for previously processed forms.

What is claimed is: 
     
         1 . A computer-implemented method of managing hierarchical forms having graphical elements, comprising:
 receiving, by a processor, an electronic image of a form;   applying, by the processor, a digital model to the electronic image, producing an identification of a list of areas within the electronic image as corresponding to a list of fields and a context for each of the list of fields in the form, the context being related to a field label and a group label in the form;   causing presenting a graphical user interface (GUI) corresponding to the form,   the GUI including a list of regions corresponding to the list of areas,   the GUI being configured for, in response to receiving a request to interact with a certain region of the list of regions corresponding to a certain area of the list of areas and a certain field of the list of fields, highlight regions of the list of regions corresponding to a context for the certain field;   receiving a correction of an identification of the certain area as corresponding to the certain field that corresponds to a certain type of graphical element, the correction indicating that the certain area does not correspond to the certain field or that the certain field has a different shape, type, or context.   
     
     
         2 . The computer-implemented method of  claim 1 ,
 the list of fields being organized hierarchically,   the field label being a label for the field and the group label being a label for a group of fields including the field.   
     
     
         3 . The computer-implemented method of  claim 1 , the applying comprising producing a confidence score for the identification of an area of the list of areas. 
     
     
         4 . The computer-implemented method of  claim 1 , the GUI further including a portion of the electronic image that does not correspond to the list of regions, the method further comprising:
 receiving a specification of a bounding box for a specific area in the portion of the electronic image;   recording an association of the specific area with a new region in the form.   
     
     
         5 . The computer-implemented method of  claim 1 , the list of regions being implemented as graphical elements, including a radio button, a check box, a drop-down list, a file upload option, or a calendar tool. 
     
     
         6 . The computer-implemented method of  claim 1 ,
 a certain region of the list of regions that corresponds to the certain area being implemented as indicators overlaid on the certain area,   the indicators comprising a bounding box and a label indicating the certain type of graphical element.   
     
     
         7 . The computer-implemented method of  claim 6 , further comprising:
 receiving an instruction to remove the bounding box or adjust a size of the bounding box;   updating information related to the correspondence between the certain area and the certain region.   
     
     
         8 . The computer-implemented method of  claim 1 , further comprising:
 receiving an indication of an interaction with a particular region corresponding to the context for the certain field;   removing the particular region from the context for the certain field.   
     
     
         9 . The computer-implemented method of  claim 1 , further comprising:
 receiving an instruction to connect the certain region and a particular region not corresponding to the context for the certain field;   adding the particular region to the context for the certain field.   
     
     
         10 . The computer-implemented method of  claim 1 , further comprising:
 receiving a specification of an action to be performed on each of the list of fields;   receiving a second electronic image of a second form from a client computer;   applying the digital model to the second electronic image, producing an identification of a second list of areas within the second electronic image as corresponding to a second list of fields and a type and a context for each of the second list of fields in the second form;   identifying, for each of the second list of fields in the second form, a corresponding field of the list of fields in the form;   determining, for a specific field of the second list of fields, a specific action to be performed based on the identifying;   causing, for the specific field, performance or an indication of performance of the specific action on the specific field.   
     
     
         11 . The computer-implemented method of  claim 10 , the identifying comprising matching a context for the field of the second list of fields with a context for the corresponding field of the list of fields. 
     
     
         12 . The computer-implemented method of  claim 11 , the matching comprising comparing a horizontal coordinate of an element of a candidate context for the field of the second list of fields with a horizontal coordinate of the field of the second list of fields. 
     
     
         13 . One or more non-transitory computer-readable storage media storing sequences of instructions which when executed cause one or more hardware processors to perform a computer-implemented method of managing hierarchical forms having graphical elements, the method comprising:
 receiving an electronic image of a form;   applying a digital model to the electronic image, producing an identification of a list of areas within the electronic image as corresponding to a list of fields and a context for each of the list of fields in the form, the context being related to a field label and a group label in the form;   causing presenting a graphical user interface (GUI) corresponding to the form,   the GUI including a list of regions corresponding to the list of areas,   the GUI being configured for, in response to receiving a request to interact with a certain region of the list of regions corresponding to a certain area of the list of areas and a certain field of the list of fields, highlight regions of the list of regions corresponding to a context for the certain field;   receiving a correction of an identification of the certain area as corresponding to the certain field that corresponds to a certain type of graphical element, the correction indicating that the certain area does not correspond to the certain field or that the certain field has a different shape, type, or context.   
     
     
         14 . The one or more non-transitory computer-readable storage media of  claim 13 , the GUI further including a portion of the electronic image that does not correspond to the list of regions, the method further comprising:
 receiving a specification of a bounding box for a specific area in the portion of the electronic image;   recording an association of the specific area with a new region in the form.   
     
     
         15 . The one or more non-transitory computer-readable storage media of  claim 13 ,
 a certain region of the list of regions that corresponds to the certain area being implemented as indicators overlaid on the certain area,   the indicators comprising a bounding box and a label indicating the certain type of graphical element.   
     
     
         16 . The one or more non-transitory computer-readable storage media of  claim 15 , the method further comprising:
 receiving an instruction to remove the bounding box or adjust a size of the bounding box;   updating information related to the correspondence between the certain area and the certain region.   
     
     
         17 . The one or more non-transitory computer-readable storage media of  claim 13 , the method further comprising:
 receiving an indication of an interaction with a particular region corresponding to the context for the certain field;   removing the particular region from the context for the certain field.   
     
     
         18 . The one or more non-transitory computer-readable storage media of  claim 13 , the method further comprising:
 receiving a specification of an action to be performed on each of the list of fields;   receiving a second electronic image of a second form from a client computer;   applying the digital model to the second electronic image, producing an identification of a second list of areas within the second electronic image as corresponding to a second list of fields and a type and a context for each of the second list of fields in the second form;   identifying, for each of the second list of fields in the second form, a corresponding field of the list of fields in the form;   determining, for a specific field of the second list of fields, a specific action to be performed based on the identifying;   causing, for the specific field, performance or an indication of performance of the specific action on the specific field.   
     
     
         19 . The one or more non-transitory computer-readable storage media of  claim 18 , the identifying comprising matching a context for the field of the second list of fields with a context for the corresponding field of the list of fields. 
     
     
         20 . The one or more non-transitory computer-readable storage media of  claim 19 , the matching comprising comparing a horizontal coordinate of an element of a candidate context for the field of the second list of fields with a horizontal coordinate of the field of the second list of fields.