Automatically identifying and interacting with hierarchically arranged elements

Abstract

A computer-implemented method of managing hierarchically arranged elements is disclosed. The method comprising: receiving, by a processor, a digital image of an electronic form having at least one group of elements, including at least one field that is programmed to receive input data; detecting, by the processor, a set of objects represented in the digital image; identifying, for each object of the set of objects, values for multiple attributes, including a type and a position, a value of the type being a group label corresponding to a group label in the electronic form, a field label corresponding to a field label in the electronic form, or a field corresponding to a field in the electronic form; building a set of feature vectors, including a feature vector for each pair of objects in the set of objects such that a first object of the pair has a type of a field and a second object of the pair has a type of a group label or a field label, the feature vector including a first feature for the type of each of the pair of objects and a second feature for the position of each of the pair of objects; determining, for each object of the set of objects having a type of a field, an associated group label and an associated field label based on the set of feature vectors; searching, for each of at least one the set of objects having a type of a field, a database for field data for the object based on the associated group label and the associated field label; causing, by the processor, displaying each of the at least one objects having a type of a field in association with the corresponding field data, thereby automatically completing the electronic form.

What is claimed is: 
     
         1 . A computer-implemented method of recognizing and completing electronic forms, comprising:
 receiving, by a processor, a digital image of an electronic form having a hierarchy of at least one group of elements, including at least one field that is programmed to receive input data;   detecting, by the processor, a set of objects represented in the digital image;   identifying, for each object of the set of objects, values for multiple attributes, including a type, a value of the type being a group label corresponding to a group label in the electronic form, a field label corresponding to a field label in the electronic form, or a field corresponding to a field in the electronic form;   building a set of feature vectors, including a first feature vector that represents a first pair of objects in the set of objects such that a first object of the first pair has a type of a field and a second object of the first pair has a type of a group label, and a second feature vector that represents a second pair of objects in the set of objects such that a first object of the second pair has a type of a field and a second object of the second pair has a type of a field label;   determining, for each object of the set of objects having a type of a field, an associated group label at a higher level of the hierarchy and an associated field label at a lower level of the hierarchy based on the set of feature vectors;   obtaining, for each of at least one of the set of objects having a type of a field, field data indicating how to fill in the field for the object based on the associated group label at a higher level of the hierarchy and the associated field label at a lower level of the hierarchy;   causing applying the field data to the electronic form, thereby automatically completing the electronic form.   
     
     
         2 . The computer-implemented method of  claim 1 , the multiple attributes further including a position, shape, font of enclosed text, or density. 
     
     
         3 . The computer-implemented method of  claim 1 ,
 the identifying comprising, for each object of the set of objects, generating encodings of the values for the multiple attributes, the set of feature vectors being built using the encodings.   
     
     
         4 . The computer-implemented method of  claim 1 , the determining further comprising:
 connecting each pair of objects in the set of objects such that one object in the pair has a type of a field and the other object in the pair has a type of a group label or a field label based on each of the multiple attributes;   combining the two objects in each of the pair of objects based on the connecting;   selecting, for each of the set of objects having a type of a field, an associated object having a type of a group label and an associated object having a type of a field label based on the combining.   
     
     
         5 . The computer-implemented method of  claim 4 , the connecting further comprising applying, for each of the pairs of objects, an instance of a digital model configured to connect and learn from the pair of objects for each of the multiple attributes. 
     
     
         6 . The computer-implemented method of  claim 4 , the combining further comprising applying, for each of the pairs of objects, an instance of a digital model configured to connect and learn from the pair of objects based on the multiple attributes. 
     
     
         7 . The computer-implemented method of  claim 4 , the selecting further comprising applying a first digital model configured to identify an object having a type of a group label and a second digital model configured to identify an object having a type of a field label. 
     
     
         8 . The computer-implemented method of  claim 7 , further comprising each of the first digital model the second digital model receiving all results of the combining as input. 
     
     
         9 . The computer-implemented method of  claim 1 , the determining further comprising:
 connecting each triplet of objects in the set of objects such that a first object in the triplet has a type of a field, a second object in the triplet has a type of a group label, and a third object in the triplet has a type of a field label based on each of the multiple attributes,   combining the three objects in each of the triplet of objects based on the connecting,   selecting, for each of the set of objects having a type of a field, an associated object having a type of a group label and an associated object having a type of a field label based on the combining.   
     
     
         10 . The computer-implemented method of  claim 1 , the determining further comprising applying, for each pair of objects in the set of objects such that one object in the pair has a type of a field and the other object in the pair has a type of a group label or a field label, an instance of a digital model configured to connect and learn from the pair of objects over the multiple attributes. 
     
     
         11 . The computer-implemented method of  claim 1 , the obtaining comprising semantically normalizing the associated group label or the associated field label. 
     
     
         12 . The computer-implemented method of  claim 1 , the field data being based on past electronic forms that were filled out. 
     
     
         13 . The computer-implemented method of  claim 1 , the field data including specific instructions for interacting with the field, including taking actions using an input device. 
     
     
         14 . One or more non-transitory storage media, storing computer-executable instructions which when executed cause one or more processors to perform a method of recognizing and completing electronic forms, the method comprising:
 receiving a digital image of an electronic form having a hierarchy of at least one group of elements, including at least one field that is programmed to receive input data;   detecting a set of objects represented in the digital image;   identifying, for each object of the set of objects, values for multiple attributes, including a type, a value of the type being a group label corresponding to a group label in the electronic form, a field label corresponding to a field label in the electronic form, or a field corresponding to a field in the electronic form;   building a set of feature vectors, including a first feature vector that represents a first pair of objects in the set of objects such that a first object of the first pair has a type of a field and a second object of the first pair has a type of a group label, and a second feature vector that represents a second pair of objects in the set of objects such that a first object of the second pair has a type of a field and a second object of the second pair has a type of a field label;   determining, for each object of the set of objects having a type of a field, an associated group label at a higher level of the hierarchy and an associated field label at a lower level of the hierarchy based on the set of feature vectors;   obtaining, for each of at least one of the set of objects having a type of a field, field data indicating how to fill in the field for the object based on the associated group label at a higher level of the hierarchy and the associated field label at a lower level of the hierarchy;   causing applying the field data to the electronic form, thereby automatically completing the electronic form.   
     
     
         15 . The one or more non-transitory storage media of  claim 14 , the determining further comprising:
 connecting each pair of objects in the set of objects such that one object in the pair has a type of a field and the other object in the pair has a type of a group label or a field label based on each of the multiple attributes;   combining the two objects in each of the pair of objects based on the connecting;   selecting, for each of the set of objects having a type of a field, an associated object having a type of a group label and an associated object having a type of a field label based on the combining.   
     
     
         16 . The one or more non-transitory storage media of  claim 15 , the connecting further comprising applying, for each of the pairs of objects, an instance of a digital model configured to connect and learn from the pair of objects for each of the multiple attributes. 
     
     
         17 . The one or more non-transitory storage media of  claim 15 , the combining further comprising applying, for each of the pairs of objects, an instance of a digital model configured to connect and learn from the pair of objects based on the multiple attributes. 
     
     
         18 . The one or more non-transitory storage media of  claim 15 , the selecting further comprising applying a first digital model configured to identify an object having a type of a group label and a second digital model configured to identify an object having a type of a field label. 
     
     
         19 . The one or more non-transitory storage media of  claim 14 , the determining further comprising:
 connecting each triplet of objects in the set of objects such that a first object in the triplet has a type of a field, a second object in the triplet has a type of a group label, and a third object in the triplet has a type of a field label based on each of the multiple attributes,   combining the three objects in each of the triplet of objects based on the connecting,   selecting, for each of the set of objects having a type of a field, an associated object having a type of a group label and an associated object having a type of a field label based on the combining.   
     
     
         20 . The one or more non-transitory storage media of  claim 14 , the determining further comprising applying, for each pair of objects in the set of objects such that one object in the pair has a type of a field and the other object in the pair has a type of a group label or a field label, an instance of a digital model configured to connect and learn from the pair of objects over the multiple attributes. 
     
     
         21 . The one or more non-transitory storage media of  claim 14 , the obtaining comprising semantically normalizing the associated group label or the associated field label.