Feedback loop for image-based recognition

Abstract

Methods, systems, and computer programs are presented for providing a feedback loop to improve object image-based recognition based on transaction data. In one method, instructions are received defining items to be visually recognized by a terminal. For each item, a check is made to determine if item image information is in a global database or if it is a new item. The global database includes item images captured during transactions performed at several terminals. For each item in the global database, item image information is downloaded from the global database. For new items, terminal cameras capture pose images for several poses of the new items, each camera taking an image for each pose. A machine-learning program is trained with the downloaded image information and the pose images, where the machine-learning program performs image-based recognition of the items that are presented at the terminal, based on images captured by the cameras.

What is claimed is: 
     
         1 . A method comprising:
 providing, by one or more processors, a user interface for selecting items to be visually recognized by a first terminal, the user interface providing options to select the items from a database coupled to a server or to select new items that are not in the database, the database comprising a plurality of items and image information that has been captured during transactions performed at the first terminal and at other terminals;   receiving, by the one or more processors, a selection of a first item to be visually recognized;   in the event of the first item is present in the database, sending, by the one or more processors, a request to the server for the image information of the first item from the database and receiving the image information from the server;   in the event of the first item is not present in the database of items, capturing, by a plurality of cameras at the first terminal, a plurality of pose images for a plurality of poses of the first item, each camera taking a pose image for each pose of the first item; and   training, by the one or more processors, a machine-learning program based on the received image information or the plurality of pose images captured at the first terminal, the machine-learning program performing image-based recognition of the selected items to be visually recognized by the first terminal based on images captured by the plurality of cameras.   
     
     
         2 . The method as recited in  claim 1 , further comprising:
 performing transactions at the first terminal after training the machine-learning program, the transactions comprising image-based recognition of items presented at the first terminal; and   transmitting image information for the performed transactions to the server for storage in the database.   
     
     
         3 . The method as recited in  claim 1 , wherein receiving the image information from the server further comprises:
 receiving the image information via a network from the server, the image information comprising images of the first item; and   storing the received image information at a local database at the first terminal.   
     
     
         4 . The method as recited in  claim 1 , wherein the image information includes one or more of image data captured by each of the plurality of cameras, a three-dimensional (3D) point cloud resulting from combining the image data from the plurality of cameras, a 3D mesh created for the item, and an item identification. 
     
     
         5 . The method as recited in  claim 1 , wherein the plurality of cameras are 3D cameras, wherein the image information includes 3D image data. 
     
     
         6 . The method as recited in  claim 1 , wherein the image-based recognition is based on an appearance of the item, wherein the image-based recognition does not include checking a Universal Product Code (UPC) code against a list of known UPC codes. 
     
     
         7 . The method as recited in  claim 1 , wherein the plurality of items includes one or more of a manufactured item, a salad, a pasta dish, a pizza box, fruit, and a vegetable. 
     
     
         8 . The method as recited in  claim 1 , wherein capturing the plurality of pose images further comprises:
 providing instruction to a user to place the first item in an examination space of the first terminal;   taking a pose image by each of the cameras; and   repeating the providing the instruction and taking a pose image by each of the cameras while changing a pose of the first item until a predetermined number of pose images have been captured.   
     
     
         9 . The method as recited in  claim 1 , wherein the database further includes pose images taken by cameras at the first terminal and cameras at the other terminals. 
     
     
         10 . The method as recited in  claim 1 , further comprising:
 before training the machine-learning program, creating additional images that are derived from the pose images, wherein the training is further based on the additional images to increase a number of available images of the first item for the training.   
     
     
         11 . A terminal comprising:
 a display;   a plurality of cameras for taking images of items placed in an examination space of the terminal;   a memory comprising instructions and a machine-learning program for performing image-based recognition of items in the examination space based on the images taken by the plurality of cameras; and   one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
 providing a user interface on the display for selecting items to be visually recognized by the terminal, the user interface providing options to select the items from a database coupled to a server or to select new items that are not in the database, the database comprising a plurality of items and image information that has been captured during transactions performed at the terminal and at other terminals; 
 receiving a selection of a first item to be visually recognized; 
 in the event of the first item is present in the database, sending a request to the server for the image information of the first item from the database and receiving the image information from the server; 
 in the event of the first item is not present in the database of items, capturing, by the plurality of cameras, a plurality of pose images for a plurality of poses of the first item when placed in the examination space, wherein each camera takes a pose image for each pose of the first item; and 
 training the machine-learning program based on the received image information or based on the plurality of pose images captured at the terminal. 
   
     
     
         12 . The terminal as recited in  claim 11 , wherein the instructions further cause the one or more computer processors to perform operations comprising:
 performing transactions at the terminal after training the machine-learning program, the transactions comprising image-based recognition of items presented at the terminal; and transmitting image information for the performed transactions to the server for storage in the database.   
     
     
         13 . The terminal as recited in  claim 11 , wherein receiving the image information from the server further comprises:
 receiving the image information via a network from the server, the image information comprising images of the first item; and   storing the received image information at a local database at the terminal.   
     
     
         14 . The terminal as recited in  claim 11 , wherein the image information includes one or more of image data captured by each of the plurality of cameras, a three-dimensional (3D) point cloud resulting from combining the image data from the plurality of cameras, a 3D mesh created for the item, and an item identification. 
     
     
         15 . The terminal as recited in  claim 11 , wherein the plurality of cameras are 3D cameras, wherein the image information includes 3D image data, wherein the image-based recognition is based on an appearance of the item, wherein the image-based recognition does not include checking a Universal Product Code (UPC) code against a list of known UPC codes. 
     
     
         16 . A non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
 providing a user interface for selecting items to be visually recognized by a first terminal, the user interface providing options to select the items from a database coupled to a server or to select new items that are not in the database, the database comprising a plurality of items and image information that has been captured during transactions performed at the first terminal and at other terminals;   receiving a selection of a first item to be visually recognized;   in the event of the first item is present in the database, sending a request to the server for the image information of the first item from the database and receiving the image information from the server;   in the event of the first item is not present in the database of items, capturing, by a plurality of cameras at the first terminal, a plurality of pose images for a plurality of poses of the first item, wherein each camera takes a pose image for each pose of the first item; and training a machine-learning program based on the received image information or the plurality of pose images captured at the first terminal, wherein the machine-learning program performs image-based recognition of the selected items to be visually recognized by the first terminal based on images captured by the plurality of cameras.   
     
     
         17 . The machine-readable storage medium as recited in  claim 16 , wherein the machine further performs operations comprising:
 performing transactions at the first terminal after training the machine-learning program, the transactions comprising image-based recognition of items presented at the first terminal; and   transmitting image information for the performed transactions to the server for storage in the database.   
     
     
         18 . The machine-readable storage medium as recited in  claim 16 , wherein receiving the image information from the server further comprises:
 receiving the image information via a network from the server, the image information comprising images of the first item; and   storing the received image information at a local database at the first terminal.   
     
     
         19 . The machine-readable storage medium as recited in  claim 16 , wherein the image information includes one or more of image data captured by each of the plurality of cameras, a three-dimensional (3D) point cloud resulting from combining the image data from the plurality of cameras, a 3D mesh created for the item, and an item identification. 
     
     
         20 . The machine-readable storage medium as recited in  claim 16 , wherein the plurality of cameras are 3D cameras, wherein the image information includes 3D image data, wherein the image-based recognition is based on an appearance of the item, wherein the image-based recognition does not include checking a Universal Product Code (UPC) code against a list of known UPC codes.