Method, a device and a system for checkout

Abstract

The present invention discloses a method, a device, and a system for checkout. Items are provided in a case with a locked door. The system and method unlock the door in response to a request from a pre-registered customer. Cameras trained on the case take a plurality of hand images of the customer and use those images to identify the item in the hand as well as whether the action is a take-up action or a put-back action. The identification of the item is done with at least one pre-trained classification model, and a second pre-trained classification model may be used to distinguish between visually similar items in the case, if the first pre-trained classification model identifies an item on a list of visually similar items. Once the item and the nature of the action are identified, a shopping list is automatically generated.

What is claimed is: 
     
         1 . A checkout method comprising the steps of:
 verifying whether a received unlocking request is sent by a pre-registered customer, and if so, issuing an unlocking instruction for opening an accommodating case in which a shelf for bearing an item is arranged;   associating the customer with a take-up action or a put-back action pertaining to an item using a camera associated with the accommodating case and trained on the shelf, the camera positioned and adapted to acquire a plurality of frames of consecutive hand images of the customer take-up or put-back the item in front of the shelf, identifying the take-up action or the put-back action based on action analysis of the plurality of frames of consecutive hand images containing the item, and identifying the item through pre-trained classification models;   generating a shopping list of the customer after identifying the take-up action or the put-back action and the item at which the take-up action or the put-back action pertains; and   performing checkout of the shopping list.   
     
     
         2 . The checkout method according to  claim 1 , wherein said identifying the take-up action or the put-back action comprises:
 acquiring the plurality of frames of consecutive hand images of the customer take-up or put-back the item in front of the shelf, and establishing a motion track for the plurality of frames of consecutive hand images containing the item on a timeline,   in case that it is detected that the motion track is an inward movement from the outside of a predetermined virtual action boundary and the item is taken in the hand, identifying the action as the put-back action;   in case that it is detected that the motion track is an outward movement from the inside of the virtual action boundary and the item is taken in the hand, identifying the action as the take-up action,   wherein the virtual action boundary is a boundary which is near the frame of the accommodating case and configured to distinguish the inside or outside of the case, the outside of the boundary in a direction away from the shelf, and the inside of the virtual action boundary is in a direction close to the shelf.   
     
     
         3 . The checkout method according to  claim 1 , wherein said identifying the item at which the take-up action or the put-back action pertains comprises the steps of:
 performing target detection on the plurality of acquired frames of hand images containing the item to obtain a plurality of rectangular area images correspondingly, wherein the rectangular area images are images corresponding to rectangular areas containing the item;   acquiring a plurality of classification results correspondingly, according to the plurality of rectangular area images and a pre-trained classification model, the pre-trained classification model is a model that is constructed by an image identification technique of a convolutional neural network and trained by all the items in the accommodating case; and   using the classification result as an identified item.   
     
     
         4 . The checkout method according to  claim 3 , said acquiring a plurality of classification results further comprising:
 acquiring a plurality of primary classification results correspondingly, according to the plurality of rectangular area images and a pre-trained first-level classification model, and acquiring a first-level classification result of the plurality of frames of hand images containing the item according to the plurality of primary classification results and a pre-trained first-level linear regression model, the classification model includes the first-level classification model trained by all the items in the accommodating case;   using the first-level classification result as a first classification result; and   using the first classification result as an identified item.   
     
     
         5 . A checkout system comprising a client terminal and a checkout device,
 the client terminal configured to receive identity information inputted by a customer upon registration and send the identity information to a checkout device, and configured to show a shopping list issued by the checkout device; and   the checkout device configured to execute the checkout method according to  claim 1 .   
     
     
         6 . A checkout system comprising a client terminal and a checkout device,
 the client terminal configured to receive identity information inputted by a customer upon registration and send the identity information to a checkout device, and configured to show a shopping list issued by the checkout device; and   the checkout device configured to execute the checkout method according to  claim 2 .   
     
     
         7 . A checkout device comprising:
 a case sized and adapted to accommodate one or more items, the case having   a shelf within the case to support the one or more items,   one or more cameras arranged within the case,   a closure that blocks access to the case when closed, and   an electronically-actuatable lock on the closure;   a registration module configured to receive identity information and account information that are inputted by a customer upon registration;   a verification association module configured to be connected with the registration module and configured to verify whether a received unlocking request is sent by a pre-registered customer, and if so, to issue an unlocking instruction for opening the lock on the closure of the case and to associate the customer with a take-up action or a put-back action pertaining to an item of the one or more items by using the one or more cameras to acquire a plurality of frames of consecutive hand images of take-up or put-back an item, said associating based on optical motion analysis of the plurality of frames of consecutive hand images containing the item to identify the take-up action or the put-back action and automated image analysis using one or more machine-implemented, pre-trained classification models to identify the item;   a shopping list generation module configured to be connected with the verification association module and configured to generate a shopping list of the customer after identifying the take-up action or the put-back action and the item at which the take-up action or the put-back action pertains; and   a checkout module configured to be connected with the shopping list generation module and configured to perform checkout of the shopping list generated by the shopping list generation module.   
     
     
         8 . The checkout device according to  claim 7 , wherein the shopping list generation module comprises:
 an action identification unit configured to acquire the plurality of frames of consecutive hand images of the customer take-up or put-back the item in front of the shelf, and establish a motion track for the plurality of frames of consecutive hand images containing the item on a timeline, in case that it is detected that the motion track is an inward movement from the outside of a predetermined virtual action boundary and the item is taken in the hand, identify the action as the put-back action; in case that it is detected that the motion track is an outward movement from the inside of the virtual action boundary and the item is taken in the hand, identify the action as the take-up action, wherein the outside of the virtual action boundary is in a direction away from the shelf, and the inside of the virtual action boundary is in a direction close to the shelf;   an item identification unit configured to identify the item at which the take-up action or the put-back action pertains; and   a shopping list generation unit configured to be connected with the verification association module, the action identification unit and the item identification unit, and configured to generate the shopping list of the customer according to the identity information of the customer associated by the verification association module, the take-up action or the put-back action identified by the action identification unit and the item at which the take-up action or the put-back action pertains identified by the item identification unit.   
     
     
         9 . The checkout device according to  claim 8 , wherein the item identification unit comprises:
 a target detection subunit configured to perform target detection on the plurality of frames of hand images containing the item that are acquired by the action identification unit, in order to obtain a plurality of rectangular area images correspondingly, wherein the rectangular area images are images corresponding to rectangular areas containing the item, and the plurality of frames of hand images containing the item correspond to a plurality of cameras in a one-to-one manner;   a first classification subunit configured to acquire a plurality of primary classification results correspondingly, according to the plurality of rectangular area images and a pre-trained first-level classification model, and configured to acquire a first-level classification result of the plurality of frames of hand images containing the item according to the plurality of primary classification results and a pre-trained first-level linear regression model, wherein the pre-trained first-level classification model is a model that is constructed by an image identification technique of a convolutional neural network and trained by the one or more items in the accommodating case;   a confirmation subunit configured to use the first-level classification result as a first classification result; and   a result determination subunit configured to use the first classification result as a to-be-identified item.   
     
     
         10 . The checkout device according to  claim 9 , wherein the item identification unit further comprises:
 a judgment subunit configured to be connected with the first classification subunit and a second classification subunit, and configured to maintain a list of similar items of the one or more items, to judge whether the first-level classification result reflects an item of the list of similar items, and to execute a second classification subunit in response to a positive judgment; and   the second classification subunit configured to obtain a plurality of secondary classification results correspondingly, according to the plurality of rectangular area images and a pre-trained second-level classification model, acquire a second-level classification result of the plurality of frames of hand images containing the item according to the plurality of secondary classification results and a pre-trained second-level linear regression model, and use the second-level classification result as a first classification result, wherein the second-level classification model is a model that is constructed by the image identification technique of the convolutional neural network and trained by the one or more items in the accommodating case in advance.   
     
     
         11 . A checkout device comprising:
 a camera configured to be arranged on an accommodating case in which a shelf for bearing an item is arranged and configured to acquire a hand image containing an item of a customer;   a processor; and   a memory that records processor-executable instructions,   wherein the processor is configured to:   verify whether a received unlocking request is sent by the pre-registered customer, if so, issue an unlocking instruction for opening the accommodating case, and associate the customer with a take-up action or a put-back action pertaining to the item by using the camera to acquire a consecutive plurality of the hand images of the customer take-up or put-back the item, said associating based on optical motion analysis of the consecutive plurality of the hand images containing the item to identify the take-up action or the put-back action and automated image analysis using one or more machine-implemented, pre-trained classification models to identify the item,   generate a shopping list of the customer after identifying the take-up action or the put-back action and the item at which the take-up action or the put-back action pertains, and   perform checkout of the shopping list.   
     
     
         12 . A checkout system comprising:
 a client terminal configured to receive identity information inputted by a customer upon registration and send the identity information to a checkout device, and configured to receive a shopping list issued by the checkout device;   an accommodating case configured to be arranged with a shelf for bearing an item, and with a door lock on a door of the accommodating case; and   the checkout device according to  claim 11 .