Importing, interpreting, and presenting data

Abstract

Implementations generally relate to importing data and presenting the data in a user interface (UI). In some implementations, a method includes capturing an image of an object using a camera, where the object includes text. The method further includes recognizing the text and recognizing data in a table. The method further includes generating a data structure that includes the text or the data in the table. The method further includes generating a graphical image that represents at least a portion of the text or the data in the table. The method further includes displaying the graphical image in a UI in a display screen of a client device.

We claim: 
     
       1. A method for interactively assisting a user to import data using a client device with a client display screen, comprising the following steps:
 receiving a raw image from the user, the raw image showing at least a part of an object that includes one or more areas with data in text format; 
 scanning the raw image to recognize data, wherein the scanning comprises:
 finding vertically aligned adjacent characters to detect words; 
 finding vertically aligned adjacent words to detect text lines; and 
 finding horizontally aligned adjacent text lines to detect one or more text blocks; 
 
 of at least one of the one or more text blocks, determining a width and a height; 
 based, at least in part, on the determined width and height, determining if the text block represents a column; and 
 finding horizontally adjacent text blocks that represent a columns to detect a first table; 
 determining coordinates for a table outline of the first table; 
 based on the coordinates, drawing the table outline in the raw image to indicate table detection; 
 based on the recognized data, proposing a target data area in the raw image to the user; 
 receiving first user input for modifying the proposed target data area; modifying the target data area based on the first user input; 
 receiving second user input indicating whether additional target areas must be added and if the additional target area is to be added, repeating the proposing of the target data area, the receiving of the first user input, the modifying of the target data area, and the receiving of the second user input; 
 upon determining that the second user input indicates that no further target areas must be added, displaying a summary of the target data areas and modified target data areas on the client display screen, and prompting the user to indicate any final adjustments that must be made; 
 receiving third user input with information about any final adjustments to be made and processing the target data areas to incorporate the final adjustments; 
 interpreting information in the target data areas to produce interpretation results; and 
 creating a graphical representation of the interpretation results including the first table, and displaying the graphical representation of the interpretation results on the client display screen. 
 
     
     
       2. The method of  claim 1 , wherein the graphical representation of the interpretation results includes a data visualization. 
     
     
       3. The method of  claim 1 , wherein recognized data includes at least one of a table, data in tabular form, a graph, or a pie chart. 
     
     
       4. The method of  claim 1 , wherein the characters includes at least one of letters, digits, symbols, emoji, characters known in any alphabet, characters known in any non-alphabetic writing system, letter arrangements in Hangeul, or Chinese characters. 
     
     
       5. The method of  claim 1 , wherein vertically aligned adjacent characters and vertically aligned adjacent words share a baseline. 
     
     
       6. The method of  claim 1 , wherein horizontally aligned adjacent text lines share a horizontal alignment, wherein the horizontal alignment includes at least one of left alignment, right alignment, or center alignment. 
     
     
       7. The method of  claim 1 , wherein determining if the text block represents a column comprises determining if the width of the text block is smaller than its height and determining if the width of the text block is smaller than an object text area width. 
     
     
       8. The method of  claim 1 , further comprising the following steps:
 detecting a table within the raw image; 
 in the table, determining a grid structure with grid cells arranged in columns and rows; 
 reading recognized data of at least part of the grid cells into a first data structure with a provisional data type; 
 determining final data types for the at least part of the grid cells; 
 confirming whether grid cells in a first row include column headers and whether grid cells in a first column include row headers; 
 creating a second data structure, with elements matching the grid structure and using the final data types, and filling the second data structure with the recognized data; 
 emphasizing the table in the raw image on the client display screen and presenting the user with an option for displaying the recognized data in a requested data visualization; 
 receiving user input requesting display of the recognized data in the requested data visualization; and 
 creating the requested data visualization and displaying it on the client display screen overlaying the raw image at a location of the emphasized table. 
 
     
     
       9. The method of  claim 8 , wherein determining a grid structure includes determining if and where the table includes inner vertical and/or horizontal border lines. 
     
     
       10. The method of  claim 8 , wherein determining a grid structure includes determining if small text blocks within individual columns are vertically aligned with small text blocks within adjacent columns. 
     
     
       11. The method of  claim 8 , wherein determining final data types for grid cells includes determining if most grid cells within a column can be characterized with a candidate first final data type and determining if most grid cells within a row can be characterized with a candidate second final data type. 
     
     
       12. The method of  claim 8 , wherein confirming whether grid cells include headers comprises determining typographical differences with other grid cells and/or determining different presentation from other grid cells. 
     
     
       13. An apparatus comprising a digital processor coupled to a client display screen and to a processor-readable non-transitory storage device, wherein the processor-readable non-transitory storage device includes one or more instructions executable by the digital processor to perform the following steps:
 receiving a raw image from a user, the raw image showing at least a part of an object that includes one or more areas with data in text format; 
 scanning the raw image to recognize data, wherein the scanning comprises: 
 finding vertically aligned adjacent characters to detect words;
 finding vertically aligned adjacent words to detect text lines; and 
 finding horizontally aligned adjacent text lines to detect one or more text blocks; 
 
 of at least one of the one or more text blocks, determining a width and a height;
 based, at least in part, on the determined width and height, determining if the text block represents a column; and 
 finding horizontally adjacent text blocks that represent a columns to detect a first table; 
 
 determining coordinates for a table outline of the first table; 
 based on the coordinates, drawing the table outline in the raw image to indicate table detection;
 based on the recognized data, proposing a target data area in the raw image to the user; 
 receiving first user input for modifying the proposed target data area; 
 modifying the target data area based on the first user input; 
 receiving second user input indicating whether additional target areas is to be added and if the additional target area is to be added, repeating the proposing of the target data area, the receiving of the first user input, the modifying of the target data area, and the receiving of the second user input; 
 upon determining that no further target areas must be added, displaying a summary of the target data areas and the modified target data areas on the client display screen, and prompting the user to indicate any final adjustments that must be made; 
 receiving third user input with information about any final adjustments to be made and processing the target data areas to incorporate the final adjustments; 
 interpreting information in the target data areas to produce interpretation results; and 
 creating a graphical representation of the interpretation results and displaying the graphical representation of the interpretation results including the first table, on the client display screen, wherein the graphical representation may include a data visualization. 
 
 
     
     
       14. The apparatus of  claim 13 , further comprising the following steps:
 detecting a table within the raw image; 
 in the table, determining a grid structure with grid cells arranged in columns and rows; 
 reading recognized data of at least part of the grid cells into a first data structure with a provisional data type; 
 determining final data types for the at least part of the grid cells; 
 confirming whether grid cells in a first row include column headers and whether grid cells in a first column include row headers; 
 creating a second data structure, with elements matching the grid structure and using the final data types, and filling the second data structure with the recognized data; 
 emphasizing the table in the raw image on the client display screen and presenting the user with an option for displaying the recognized data in a requested data visualization; 
 receiving user input requesting display of the recognized data in the requested data visualization; and 
 creating the requested data visualization and displaying it on the client display screen overlaying the raw image at a location of the emphasized table. 
 
     
     
       15. A non-transitory computer-readable storage medium carrying program instructions thereon, the instructions when executed by one or more processors cause the one or more processors to perform operations comprising the following steps:
 receiving a raw image from a user, the raw image showing at least a part of an object that includes one or more areas with data in text format; 
 scanning the raw image to recognize data, wherein the scanning comprises:
 finding vertically aligned adjacent characters to detect words; 
 finding vertically aligned adjacent words to detect text lines; and 
 finding horizontally aligned adjacent text lines to detect one or more text blocks; 
 of at least one of the one or more text blocks, determining a width and a height; 
 based, at least in part, on the determined width and height, determining if the text block represents a column; and 
 finding horizontally adjacent text blocks that represent a columns to detect a first table; 
 
 determining coordinates for a table outline of the first table; 
 based on the coordinates, drawing the table outline in the raw image to indicate table detection;
 based on recognized data, proposing a target data area in the raw image to the user; 
 receiving first user input for modifying the proposed target data area, and modifying the target data area based on the first user input; 
 receiving second user input indicating whether additional target areas must be added and if the additional target area is to be added, repeating the proposing of the target data area, the receiving of the first user input, the modifying of the target data area, and the receiving of the second user input; 
 upon determining that no further target areas must be added, displaying a summary of the target data areas and the modified target data areas on a client display screen, and prompting the user to indicate any final adjustments that must be made; 
 receiving third user input with information about any final adjustments to be made and processing the target data areas to incorporate the final adjustments; 
 interpreting information in the target data areas to produce interpretation results; and 
 creating a graphical representation of the interpretation results and displaying the graphical representation of the interpretation results including the first table, on the client display screen, wherein the graphical representation may include a data visualization. 
 
 
     
     
       16. The storage medium of  claim 15 , wherein the operations further comprise the following steps:
 detecting a second table within the raw image; 
 in the second table, determining a grid structure with grid cells arranged in columns and rows; 
 reading recognized data of grid cells into a first data structure with a provisional data type; 
 determining final data types for grid cells; 
 confirming whether grid cells in a first row of the rows include column headers and whether grid cells in a first column include row headers; 
 creating a second data structure, with elements matching the grid structure and using the final data types, and filling the second data structure with the recognized data; 
 emphasizing the second table in the raw image on the client display screen and presenting the user with an option for displaying the recognized data in a requested data visualization; 
 receiving user input requesting display of the recognized data in the requested data visualization; and 
 creating the requested data visualization and displaying it on the client display screen overlaying the raw image at a location of the emphasized second table.