Cashier interface for linking customers to virtual data

Abstract

A system and method for applications of computer vision in linking users with virtual data that can include detecting digital interaction state of a plurality of subjects in an environment using at least one sensor-based monitoring system; detecting a contextual organization of subjects relative to an operator station; at the operator station, augmenting the user interface based on the contextual organization of subjects which comprises of at least: presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization, and in response to received user interaction with at least one selected subject indicator, accessing the digital interaction state of the subject associated the at least one subject indicator.

We claim: 
     
         1 . A method comprising:
 detecting digital interaction state of a plurality of subjects in an environment using at least one sensor-based monitoring system;   detecting a contextual organization of subjects relative to an operator station;   at the operator station, augmenting the user interface based on the contextual organization of subjects which comprises of at least:
 presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization, and 
 in response to received user interaction with at least one selected subject indicator, accessing the digital interaction state of the subject associated the at least one subject indicator. 
   
     
     
         2 . The method of  claim 1 , wherein detecting digital interaction state comprises of: tracking a set of subjects through the environment and, for each subject, detecting item interaction events including at least item selection events and updating items in a checkout list based on the item interaction event; and wherein accessing the digital interaction state of the subject comprises accessing the checkout list of the subject an associated with the at least one subject indicator. 
     
     
         3 . The method of  claim 2 , wherein accessing the checkout list of the subject comprises executing a checkout process for items of the checkout list. 
     
     
         4 . The method of  claim 3 , wherein accessing the checkout list of the subject further comprises presenting a representation of the checkout list within the user interface. 
     
     
         5 . The method of  claim 3 , wherein accessing the checkout list of the subject further comprises, if a cart issue is associated with the checkout list, presenting a guided resolution interaction flow within the user interface to resolve the cart issue prior to executing the checkout process. 
     
     
         6 . The method of  claim 3 , wherein detecting contextual organization of subjects may include detecting social grouping of subjects and associating multiple subjects with a single checkout list. 
     
     
         7 . The method of  claim 2 , wherein accessing the checkout list of the subject comprises adding at least one item to the checkout list of the subject based on received user interaction at the operator station. 
     
     
         8 . The method of  claim 2 , wherein detecting item interaction events comprises of collecting image data, applying computer vision processing of the image data and detecting item interaction events in part from the computer vision processing. 
     
     
         9 . The method of  claim 2 , wherein detecting item interaction events comprises of detecting user-item interactions using computer vision processing of image data and a smart shelf event data. 
     
     
         10 . The method of  claim 2 , wherein detecting contextual organization comprises detecting line order by analyzing orientation and position of subjects relative to the operator station. 
     
     
         11 . The method of  claim 10 , wherein detecting line order further comprises detecting a direction of attention of one of the operator station or an operator. 
     
     
         12 . The method of  claim 10 , wherein presenting a set of subject indicators in the user interface with the subject indicators arranged in response to contextual organization comprises ordering the subject indicators in an order corresponding to the line order. 
     
     
         13 . The method of  claim 1 , wherein presenting a set of subject indicators in the user interface comprises generating an interactive image-based representation of the position of subjects relative to the operator station and presenting the image-based representation with selectable subject indicators at the location of the subjects represented in the image-based representation. 
     
     
         14 . The method of  claim 13 , wherein presenting the image-based representation comprises rendering the image-based representation in an augmented reality display. 
     
     
         15 . A system comprising:
 a sensor-based monitoring system comprising at least a computer vision monitoring system with a set of imaging devices;   a computing device at an operator station, the computing device operating a subject management user interface; and   one or more computer-readable mediums in communication with the sensor-based monitoring system and computing device, the one or more computer readable mediums storing instructions that, when executed by the one or more computer processors, cause the computer processors to:
 detect user-associated data of a plurality of subjects in an environment, 
 detect a contextual organization of subjects relative to the operator station, and 
 at the computing device, augment the subject management user interface based on the contextual organization of subjects, which includes presenting subject indicators in the user interface with the subject indicators arranged in response to contextual organization, and, in response to user interaction with a subject indicator, accessing the user-associated data of a subject associated with the subject indicator. 
   
     
     
         16 . The system of  claim 1 , wherein instructions to detect user-associated data comprise of: tracking a set of subjects through the environment and, for each subject, detecting item interaction events including at least item selection events and updating items in a checkout list based on the item interaction event; and wherein accessing the user-associated data of the subject comprises accessing the checkout list of the subject an associated with the at least one subject indicator. 
     
     
         17 . The system of  claim 16 , wherein detecting item interaction events comprises of collecting image data, applying computer vision processing of the image data and detecting item interaction events in part from the computer vision processing. 
     
     
         18 . The system of  claim 16 , wherein detecting item interaction events comprises of detecting user-item interactions using computer vision processing of image data and a smart shelf event data. 
     
     
         19 . The system of  claim 16 , wherein instructions to detect contextual organization comprise detecting line order by analyzing orientation and position of subjects relative to the operator station. 
     
     
         20 . The system of  claim 16 , wherein accessing the checkout list of the subject comprises executing a checkout process for items of the checkout list.