Artificial intelligence systems and methods for cashierless tracking and feedback

Abstract

A method for providing real-time recommendations to user in a shopping environment involves sampling a shopping environment using video cameras to generate video features related to a shopper in connection to an item, the sampling input to a machine learning model to create labels related to a state of a scenario, the scenario including the shopper handling the item. Supplemental information is provided to the shopper in connection with the item. The makeup of the supplemental information may be sourced from online service or device associated with the shopper. The supplemental information may be delivered to the shopper, or to a shopper-aware display in the store, or elsewhere. A processing entity associated with the store detects a scenario to identify the shopper as having finished shopping, and causing a charge of the item to a cashierless shopping cart associated with the shopper. In one example, passive, wireless weight sensors may be used to supplement input features for inferences made by the machine learning model. In another example, wireless, battery-free displays may be coupled to shelves to provide information related to supplemental information relating the item held by the shopper.

What is claimed is: 
     
         1 . A system for cashierless shopping, comprising:
 a location identified for a store to sell items;   an access into the location to monitor entry and exit of a shopper from the location;   a plurality of cameras arranged in the location to track the shopper when in the store and interactions by the shopper with one or more of said items in the store;   one or more compute nodes being local to the location for processing data captured from one or more of the plurality of cameras, wherein the local compute nodes are configured to produce data regarding a take of one or more of the items and associating the take to the shopper;   a network connection for connecting the one or more compute nodes of the location to a cloud processing system, the cloud processing system including one or more servers configured to further process the data regarding the take of one or more of the items said associating of the take of the item by the shopper, wherein the one or more servers are configured to process machine learning algorithms to learn characteristics of the interactions by the shopper in regard to said take of one or more items and said associating the take to the shopper; and   an exit sensor, the exit sensor is configured to be monitored to determine if the shopper leaves the location;   wherein said one or more servers of the cloud processing system is configured to send the shopper an electronic receipt showing the items charged to the shopper.   
     
     
         2 . The system of  claim 1 , wherein the location includes one or more other sensors in addition to the plurality of cameras, the sensors produce sensor data that is processed by the one or more compute nodes and used in said producing of data regarding the take of one or more of the items and associating the take to the shopper. 
     
     
         3 . The system of  claim 2 , wherein said one or more servers of the cloud processing system is configured to process sensor fusion to assist in verifying the take of the one or more items. 
     
     
         4 . The system of  claim 1 , further comprising an entry sensor, the entry sensor is used for the monitoring of entry of the shopper into the location of the store. 
     
     
         5 . The system of  claim 4 , wherein the entry sensor is configured to capture one or more of credit card information, mobile device information, mobile device screen data, or biometric information associated with the shopper. 
     
     
         6 . The system of  claim 4 , wherein the entry sensor is configured to capture information obtained from a mobile device of the shopper. 
     
     
         7 . The system of  claim 6 , wherein the mobile device of the shopper is configured to execute an application that relates to shopping at the location, the application is configured to provide information regarding entry, items that can be purchased, items added to an electronic cart of the shopper, and charges to be made or made to an account of the shopper. 
     
     
         8 . The system of  claim 6 , wherein mobile device executes an application associated with the store, the application is configured to render information regarding items picked for purchase, items charged to an account of the shopper and exchange of data with said one or more servers of the cloud processing system. 
     
     
         9 . The system of  claim 8 , wherein the application is further configured to render information regarding directions for navigating the store for finding one or more additional items or related to a diet of the shopper. 
     
     
         10 . The system of  claim 1 , wherein a machine learning model is used to predict one or more items for purchase by the shopper, wherein a notification to an application of a mobile of the shopper provides a recommendation regarding said items predicted for purchase. 
     
     
         11 . The system of  claim 1 , wherein the plurality of cameras are installed overhead of the location to track movement of the shopper and differentiate other shoppers in the location during said tracking of movement. 
     
     
         12 . The system of  claim 1 , wherein said machine learning algorithms process multiple scenarios including the shopper handling the item and other items to reinforce the learning of when the handling is the take of the item or a return of the item. 
     
     
         13 . The system of  claim 1 , wherein the location includes one or more additional sensors, the one or more additional sensors include weight sensors, or motion sensors, or light sensors, or audio sensors, or combinations of two or more thereof, wherein said additional sensors producing sensor input features processed by the machine learning algorithms to reinforce learning associated with identification of the item, handling of the item, movement of the item, or return of the item by the shopper. 
     
     
         14 . The system of  claim 1 , wherein items in the store are imaged and associated with a location in the store, wherein said imaging of the items assists in improving a determination of the take of the items by the shopper being tracked and a determination of a type of items identified as taken by the shopper. 
     
     
         15 . The system of  claim 1 , wherein the access is a designated entry providing an identification capturing system, and said location of the store is within another store or venue. 
     
     
         16 . The system of  claim 1 , wherein in the identification capturing system includes one or more of a credit card reader, or a mobile device detector, or an image scanner, or an image detector for capturing a screen of a mobile device, or a biometric scanner, or a combination of two or more thereof. 
     
     
         17 . The system of  claim 1 , wherein image data from the plurality of cameras is processed for computer vision processing, the computer vision processing assists in identifying one or more poses of the shopper interacting with the items in the store, wherein the one or more poses are associated with limb tracking of the shopper. 
     
     
         18 . The system of  claim 1 , wherein the location is at a stadium for purchasing food or goods using the system for cashierless shopping. 
     
     
         19 . The system of  claim 1 , wherein the exit sensor, in conjunction with data obtained from the one or more cameras, produces data monitored to determine if the shopper leaves the location with said one or more items. 
     
     
         20 . The system of  claim 1 , wherein image data from the plurality of cameras is processed for computer vision processing, the computer vision processing assists in said tracking of the shopper, or a location of the shopper, or a recognition of a type of said item or items, or items in the store, or a dwell time of a shopper, or a gaze direction of the shopper, or limb tracking of the shopper, or a combination of two or more thereof.