Occluded item detection for vision-based self-checkouts

Abstract

Item recognition of a given item is trained on a single item from different views. The item recognition is then trained on images of the given item partially occluded by a second item having same, similar, or different shapes and features to that of the given item. General features of the item are noted and used to detect the given item when the given item is presented with multiple different items having multiple different occluded views.

The invention claimed is: 
     
       1. A method, comprising:
 obtaining, by a processor, an image taken of a transaction area for multiple items present together in the image; 
 identifying, by the processor, first item identifiers for first items associated with the multiple items from first features extracted from the image, wherein the first items are not occluded within the image; 
 identifying, by the processor, second item identifiers for second items associated with the multiple items from second features extracted from the image, wherein the second items are occluded within the image; and 
 providing, by the processor, the first item identifiers and the second item identifiers during a transaction at a terminal for the multiple items. 
 
     
     
       2. The method of  claim 1 , wherein identifying the first item identifiers further includes cropping pixels out of the image for each first item creating a plurality of item images from the image, extracting the corresponding first features from each item image, scoring the first features for each item image, matching a score for each item image to a corresponding first item identifier. 
     
     
       3. The method of  claim 1 , wherein identifying the first item identifiers further includes providing the image as input to a first machine-learning algorithm that identifies the first features and provides the first item identifiers for the first items as output. 
     
     
       4. The method of  claim 3 , wherein providing the image as input further includes providing pixel coordinates for each first item with the image to the first machine-learning algorithm. 
     
     
       5. The method of  claim 4 , wherein identifying the second item identifiers further includes providing the image as input to a second machine-learning algorithm that identifies the second features and provides the second item identifiers for the second items as output. 
     
     
       6. The method of  claim 5 , wherein providing the image as input to the second machine-learning algorithm further includes providing pixel coordinates for a portion of each second item that is not occluded within the image to the second machine-learning algorithm. 
     
     
       7. The method of  claim 1 , wherein identifying the first item identifiers further includes identifying the first items from the image when pixels associated with each first item within the image is non overlapping with other pixels associated with remaining items within the image. 
     
     
       8. The method of  claim 1 , wherein identifying the second item identifiers further includes identifying the second items from the image when a portion of pixels associated each second item within the image is overlapped by other pixels associated with at least one remaining item within the image. 
     
     
       9. The method of  claim 1 , wherein providing further includes providing the first item identifiers and the second item identifiers to a transaction manager of the terminal, wherein the transaction manager processes the transaction from the image of the transaction area. 
     
     
       10. The method of  claim 1 , wherein providing further includes obtaining item details and item pricing using the first item identifiers and the second item identifiers and providing the item details and the item pricing for the first item identifiers and the second item identifiers to a transaction manager of the terminal, wherein the transaction manager processes the transaction from the image of the transaction area. 
     
     
       11. A method, comprising:
 obtaining, by a processor, a single image of a transaction area depicting multiple items during a transaction at a terminal; 
 creating, by the processor, cropped images for each item identified in the single image, wherein each cropped image having a non-occluded view of a corresponding item; 
 creating, by the processor, cropped item pair images for each pair of items identified in the single image, each cropped item pair image having an occluded view for at least one item in the corresponding pair of items; 
 providing, by the processor, the cropped images to a first machine-learning algorithm and receiving first item identifiers associated with first items represented within the single image as output; 
 providing, by the processor, the cropped item pair images to a second machine-learning algorithm and receiving the second item identifiers associated with second items represented within the single image as output; and 
 providing, by the processor, the first item identifiers and the second item identifiers to a transaction manager of the terminal to process the transaction from the single image. 
 
     
     
       12. The method of  claim 11 , wherein obtaining further includes training the first machine-learning algorithm on single item images for single items before obtaining the single image of the transaction area. 
     
     
       13. The method of  claim 12 , wherein obtaining further includes training the second machine-learning algorithm on item pair images for pairs of items that are occluded within the corresponding item pair image before obtaining the single image of the transaction area. 
     
     
       14. The method of  claim 11 , wherein creating cropped images further includes removing background pixels associated with a background of the transaction area from each cropped image. 
     
     
       15. The method of  claim 14 , wherein creating the cropped item pair images further includes removing the background pixels from each cropped item pair image. 
     
     
       16. The method of  claim 11 , wherein creating cropped images further includes identifying each item from the single image based on lines, edges, shapes, and colors detected in the single image for the corresponding item to produce each cropped image. 
     
     
       17. The method of  claim 11 , wherein creating cropped item pair images further includes placing bounding boxes around each pair of items that have occluded views detected in the single image and using the bounding box for defining the cropped item pair images. 
     
     
       18. A system, comprising:
 a camera; 
 a terminal; and 
 a server comprising a processor and a memory and in communication with the camera and the terminal, the processor and the memory configured to perform operations comprising:
 obtaining a single image captured by the camera of a transaction area associated with the terminal during a transaction at the terminal; 
 creating cropped images for first items identified within the single image, each first item is not associated with an occluded view within the single image; 
 creating cropped item pair images for each pair of second items identified within the single image, each pair of second items associated with an occluded view within the single image; 
 obtaining first item identifiers for the first items based on the cropped images; 
 obtaining second item identifiers for the second items based on the cropped item pair images; and 
 providing the first item identifiers and the second item identifiers to the terminal for processing the transaction from the single image captured by the camera. 
 
 
     
     
       19. The system of  claim 18 , wherein the terminal is a self-service terminal. 
     
     
       20. The system of  claim 18 , wherein the operations associated with obtaining the first item identifiers further includes providing the cropped images to a first machine-learning algorithm and receiving the first item identifiers as output from the first machine-learning algorithm, and wherein the operations associated with obtaining the second item identifiers further includes providing the cropped item pair images to a second machine-learning algorithm and receiving the second item identifiers as output from the second machine-learning algorithm.