Financial management using augmented reality systems

Abstract

The disclosed embodiments provide an augmented reality system. The augmented reality system includes a camera that captures a set of images of one or more physical documents. The augmented reality system also includes a processing apparatus that enables use of the augmented reality system by a user in performing financial management associated with data in the one or more physical documents.

What is claimed is: 
     
       1. A method for using an augmented reality system, comprising:
 displaying an image of an augmented reality environment including a physical document on a display device of the augmented reality system based on image data captured by a camera of the augmented reality system; 
 overlaying on the display device, while displaying the image of the augmented reality environment including the physical document, a first user interface element comprising a set of command options for interacting with the physical document in the augmented reality environment; 
 receiving a gesture via the display device, wherein:
 the display device is a touch-sensitive display device, and 
 the gesture is received via the touch-sensitive display device; 
 
 associating the gesture with a command option of the set of command options overlaid on the display device; and 
 in response to receiving the gesture:
 determining, by a processor of the augmented reality system, a digital representation of one or more parts of the physical document based on the image of the physical document displayed in the augmented reality environment; 
 performing, by the processor, an operation related to the digital representation based on the command option; and 
 displaying a result of the operation on the display device. 
 
 
     
     
       2. The method of  claim 1 , further comprising:
 receiving, via the camera, new image data of the physical document in the augmented reality environment, wherein the new image data comprises one or more new parts of the physical document; and 
 updating the digital representation based on the new image data. 
 
     
     
       3. The method of  claim 1 , further comprising:
 selecting, based on the command option, one or more data elements in the digital representation of the physical document displayed in the augmented reality environment, 
 wherein the operation performed by the processor is based on the selected one or more data elements. 
 
     
     
       4. The method of  claim 1 , wherein:
 the command option comprises a command to identify the physical document displayed in the augmented reality environment, 
 the operation comprises determining a document type associated with the physical document based on one or more of text and features of the physical document, and 
 the result of the operation comprises displaying a second user interface element on the display device comprising a document type associated with the physical document. 
 
     
     
       5. The method of  claim 1 , wherein:
 the command option comprises a command to generate a data visualization based on the physical document displayed in the augmented reality environment, 
 the operation comprises generating the data visualization based the digital representation of the physical document displayed in the augmented reality environment, and 
 the result of the operation comprises displaying the data visualization on the display device. 
 
     
     
       6. The method of  claim 1 , wherein:
 the digital representation comprises one or more data elements related to the physical document displayed in the augmented reality environment, and 
 the method further comprises:
 determining a position on the physical document displayed in the augmented reality environment corresponding to one of the one or more data elements in the digital representation; 
 receiving, via the camera, new image data of a second physical document in the augmented reality environment; and 
 determining a digital data value for a data element based on text in the second physical document at the determined position. 
 
 
     
     
       7. The method of  claim 1 , wherein the augmented reality system comprises one of a mobile phone or a tablet computer. 
     
     
       8. The method of  claim 1 , wherein the gesture comprises a selection of one or more data elements of the physical document. 
     
     
       9. The method of  claim 8 , wherein performing the operation related to the digital representation comprises performing an action with respect to the one or more data elements. 
     
     
       10. An augmented reality system, comprising:
 a camera configured to capture image data; 
 a display device configured to display image data; and 
 a processor in data communication with the camera and the display device, and configured to cause the augmented reality system to:
 display an image of an augmented reality environment including a physical document on the display device based on image data captured by the camera; 
 overlay on the display device, while displaying the image of the augmented reality environment including the physical document, a first user interface element comprising a set of command options for interacting with the physical document in the augmented reality environment; 
 receive a gesture via the display device, wherein:
 the display device is a touch-sensitive display device, and 
 the gesture is received via the touch-sensitive display device; 
 
 associate the gesture with a command option of the set of command options overlaid on the display device; and 
 in response to receiving the gesture:
 determine a digital representation of one or more parts of the physical document based on the image of the physical document displayed in the augmented reality environment; 
 perform an operation related to the digital representation based on the command option; and 
 display a result of the operation on the display device. 
 
 
 
     
     
       11. The augmented reality system of  claim 10 , wherein the processor is further configured to cause the augmented reality system to:
 receive, via the camera, new image data of the physical document in the augmented reality environment, wherein the new image data comprises one or more new parts of the physical document; and 
 update the digital representation based on the new image data. 
 
     
     
       12. The augmented reality system of  claim 10 , wherein the processor is further configured to cause the augmented reality system to:
 select, based on the command option, one or more data elements in the digital representation of the physical document displayed in the augmented reality environment, 
 wherein the operation performed by the processor is based on the selected one or more data elements. 
 
     
     
       13. The augmented reality system of  claim 10 , wherein:
 the command option comprises a command to identify the physical document displayed in the augmented reality environment, 
 the operation comprises determining a document type associated with the physical document based on one or more of text and features of the physical document, and 
 the result of the operation comprises displaying a second user interface element on the display device comprising a document type associated with the physical document. 
 
     
     
       14. The augmented reality system of  claim 10 , wherein:
 the command option comprises a command to generate a data visualization based on the physical document displayed in the augmented reality environment, 
 the operation comprises generating the data visualization based the digital representation of the physical document displayed in the augmented reality environment, and 
 the result of the operation comprises displaying the data visualization on the display device. 
 
     
     
       15. The augmented reality system of  claim 10 , wherein:
 the digital representation comprises one or more data elements related to the physical document displayed in the augmented reality environment, and 
 the processor is further configured to:
 determine a position on the physical document displayed in the augmented reality environment corresponding to one of the one or more data elements in the digital representation; 
 receive, via the camera, new image data of a second physical document in the augmented reality environment; and 
 determine a digital data value for a data element based on text in the second physical document at the determined position. 
 
 
     
     
       16. The augmented reality system of  claim 10 , wherein the augmented reality system comprises one of a mobile phone or a tablet computer. 
     
     
       17. The augmented reality system of  claim 10 , wherein the gesture comprises a selection of one or more data elements of the physical document. 
     
     
       18. The augmented reality system of  claim 17 , wherein performing the operation related to the digital representation comprises performing an action with respect to the one or more data elements. 
     
     
       19. A method for using an augmented reality system, comprising:
 displaying an image of an augmented reality environment including a physical document on a display device of the augmented reality system based on image data captured by a camera of the augmented reality system; 
 overlaying on the display device, while displaying the image of the augmented reality environment including the physical document, a first user interface element comprising a set of command options for interacting with the physical document in the augmented reality environment; 
 receiving a gesture via the display device, wherein:
 the display device is a touch-sensitive display device, and 
 the gesture comprises circling or highlighting on or more data elements of the physical document via the touch-sensitive display device; 
 
 associating the gesture with a command option of the set of command options overlaid on the display device; and 
 in response to receiving the gesture:
 determining, by a processor of the augmented reality system, a digital representation of one or more parts of the physical document based on the image of the physical document displayed in the augmented reality environment; 
 performing, by the processor, an operation related to the digital representation based on the command option; and 
 displaying a result of the operation on the display device. 
 
 
     
     
       20. The method of  claim 19 , further comprising:
 receiving, via the camera, new image data of the physical document in the augmented reality environment, wherein the new image data comprises one or more new parts of the physical document; and 
 updating the digital representation based on the new image data.