Identifying location of shreds on an imaged form

Abstract

Disclosed herein is a machine learning application for automatically reading filled-in forms. There are multiple steps involved in using a computer to accurately read a handwritten form. First, the system identifies the form. Second, the system identifies what parts of the form are important. Third, the important parts are extracted as image data (known as shreds). Finally, fourth, the system interprets the shreds. This application is focused on steps two and three of that overall process. The disclosed techniques relate to training a machine learning system on a given series of forms such that when provided future filled-in forms within that series, the system is able to extract the portions of the filled-in form that are important/relevant.

1 . A method configured to identify locations of fields of a given form template comprising:
 receiving image data of a plurality of filled-in forms having a given form type;   calculating an average pixel set across all of the plurality of filled-in forms;   generating reconstructed image data of each of the plurality of filled-in forms from the average pixel set;   generating reduced image data by subtracting the reconstructed image data from the image data on a reconstructed form to corresponding original form basis, wherein the reduced image data includes a plurality of reduced forms;   determining pixel values comprising a pixel value for each pixel location in the reduced image data, wherein each pixel value of the pixel values represents a magnitude of variance of a corresponding pixel location in the reduced image data;   identifying a set of pixel regions of the pixel values that include pixels above a threshold of the magnitude of variance; and   extracting, based on said identifying, image shreds from the image data at pixel coordinates corresponding to the set of pixel regions.   
     
     
         2 . The method of  claim 1 , further comprising:
 shifting the image data into greyscale.   
     
     
         3 . The method of  claim 1 , further comprising:
 inverting pixel values of the image data.   
     
     
         4 . The method of  claim 1 , further comprising:
 blurring the plurality of reduced forms.   
     
     
         5 . The method of  claim 1 , wherein said identifying the set of pixel regions is performed via identification of connected components. 
     
     
         6 . The method of  claim 1 , further comprising:
 blurring the reduced image data.   
     
     
         7 . The method of  claim 1 , wherein calculating a plurality of coordinate sets of input fields comprises:
 identifying connected components of pixel values in the reduced image data; and   calculating, based on the connected components, the plurality of coordinate sets of input fields.   
     
     
         8 . A system comprising:
 a processor; and   a memory having instructions configured to cause the processor to identify coordinate regions corresponding to a set of input fields on image data of a plurality of filled-in forms all matching a given form type, wherein the image data is reduced through subtraction of pixels that exist across the plurality of filled-in forms and the coordinate regions are identified as connected components from a set of remaining pixels after the subtraction.   
     
     
         9 . The system of  claim 8 , wherein pixels that exist across the plurality of filled-in forms are identified with a modified set of individual form images that are reconstructed from an average pixel set across all of the plurality of filled-in forms. 
     
     
         10 . The system of  claim 9 , wherein the image data is reduced by subtracting the set of modified individual form images form the plurality of filled-in forms on a reconstructed form to corresponding original form basis. 
     
     
         11 . The system of  claim 8 , wherein the memory further includes instructions configured to cause the processor to extract image shreds from each of the plurality of filled-in forms according to the coordinate regions. 
     
     
         12 . The system of  claim 8 , wherein the memory further includes instructions configured to cause the processor to shift the image data into greyscale. 
     
     
         13 . The system of  claim 9 , wherein the memory further includes instructions configured to cause the processor to invert pixel values of the image data before identifying the average pixel set across all of the plurality of filled-in forms. 
     
     
         14 . A system configured to identify locations of fields of a given form template comprising:
 a processor; and   a memory having instructions configured to cause the processor to:
 receive image data of a plurality of filled-in forms having a given form type; 
 calculate an average pixel set across all of the plurality of filled-in forms; 
 generating reconstructed image data of each of the plurality of filled-in forms from the average pixel set; 
 generate reduced image data by subtracting the reconstructed image data from the image data on a reconstructed form to corresponding original form basis, wherein the reduced image data includes a plurality of reduced forms; 
 determine pixel values comprising a pixel value for each pixel location in the reduced image data, wherein each pixel value of the pixel values represents a magnitude of variance of a corresponding pixel location in the reduced image data; 
 identify a set of pixel regions of the pixel values that include pixels above a threshold of the magnitude of variance; and 
 extract, based on said identifying, image shreds from the image data at pixel coordinates corresponding to the set of pixel regions. 
   
     
     
         15 . The system of  claim 14 , the instructions further comprising:
 shifting the image data into greyscale.   
     
     
         16 . The system of  claim 14 , the instructions further comprising:
 inverting pixel values of the image data.   
     
     
         17 . The system of  claim 14 , the instructions further comprising:
 blurring the plurality of reduced forms.   
     
     
         18 . The system of  claim 14 , wherein said identifying the set of pixel regions is performed via identification of connected components. 
     
     
         19 . The system of  claim 14 , further comprising:
 blurring the reduced image data.   
     
     
         20 . The system of  claim 14 , wherein calculating a plurality of coordinate sets of input fields comprises:
 identifying connected components of pixel values in the reduced image data; and   calculating, based on the connected components, the plurality of coordinate sets of input fields.