3D product reconstruction from multiple images collected at checkout lanes

Abstract

Techniques for three-dimensional product reconstruction using multiple images collected at checkout lanes are disclosed herein. An example method includes capturing, by a barcode reader associated with a point of sale (POS) workstation, first image data associated with each of a plurality of products passing through a product scanning region of the POS workstation; analyzing barcode data from the first image data captured by the barcode reader to identify each product of the plurality of products passing through the product scanning region of the POS workstation; capturing, by one or more color cameras associated with the POS workstation, second image data associated with the identified product, of the plurality of products passing through the product scanning region of the POS workstation; and generating, by a processor, a textured three-dimensional mesh reconstruction for the identified product based on the second image data associated with the identified product.

The invention claimed is: 
     
       1. A method for three-dimensional product reconstruction using multiple images collected at POS workstations, comprising:
 capturing, by a barcode reader associated with a point of sale (POS) workstation, first image data associated with a plurality of products passing through a product scanning region of the POS workstation; 
 analyzing barcode data from the first image data captured by the barcode reader to identify at least some of the plurality of products passing through the product scanning region of the POS workstation; 
 responsive to identifying a product, from the plurality of products, resulting in an identified product, capturing, by one or more color cameras associated with the POS workstation, second image data associated with the identified product; and 
 generating, by a processor, a textured three-dimensional mesh reconstruction for the identified product based on the second image data associated with the identified product, 
 wherein generating the textured three-dimensional mesh reconstruction associated with the identified product based on the second image data associated with the identified product includes:
 for each image of the second image data associated with the identified product; 
 segmenting, by the processor, the identified product from the background, or any hands in the image; 
 analyzing, by the processor, the segmented image of the identified product to determine a depth and a texture associated with each pixel of the identified product in the image; and 
 generating, by the processor, a three-dimensional point cloud associated with the segmented image of the identified product based on the depth and the texture associated with each pixel of the identified product; and 
 
 stitching, by the processor, the three-dimensional point clouds associated with each image of the second image data associated with the identified product together to generate a textured three-dimensional mesh reconstruction associated with the identified product, and 
 wherein stitching, by the processor, the three-dimensional point cloud associated with each image of the second image data associated with the identified product together to generate a textured three-dimensional mesh reconstruction associated with the product includes:
 for each three-dimensional point cloud associated with the identified product:
 identifying, by the processor, a location of a barcode or other product label in the three-dimensional point cloud; and 
 determining, by the processor, an orientation of the product in the three-dimensional point cloud based on the identified location of the barcode or other product label in the three-dimensional point cloud; and 
 
 
 stitching, by the processor, the three-dimensional point clouds associated with each image of the second image data associated with the identified product together to generate the textured three-dimensional mesh reconstruction associated with the product, based on the determined orientation of the product in each three-dimensional point cloud. 
 
     
     
       2. The method of  claim 1 , wherein generating the textured three-dimensional mesh reconstruction for the identified product based on the second image data associated with the identified product further includes:
 determining, by the processor, an indication of a shape associated with the identified product; 
 generating, by the processor, a three-dimensional mesh associated with the identified product based on the indication of the shape associated with the identified product; and 
 stitching, by the processor, the second image data associated with the identified product to the three-dimensional mesh associated with the identified product in order to generate a textured three-dimensional mesh associated with the identified product. 
 
     
     
       3. The method of  claim 2 , wherein determining the indication of the shape associated with the identified product includes:
 accessing, by the processor, a database storing an indication of a product shape associated with the identified product. 
 
     
     
       4. The method of  claim 2 , wherein determining the indication of the shape associated with the product shown in the image includes:
 for at least one image of the second image data associated with the identified product:
 segmenting, by the processor, the identified product from the background, or any hands in the image; 
 analyzing, by the processor, the segmented image of the identified product using a neural network to classify the shape associated with the identified product as one of a plurality of possible product shapes. 
 
 
     
     
       5. The method of  claim 2 , wherein stitching the second image data associated with the identified product to the three-dimensional mesh associated with the identified product in order to generate the textured three-dimensional mesh associated with the identified product includes:
 for each image of the second image data associated with the identified product:
 segmenting, by the processor, the identified product from the background, or any hands in the image; 
 identifying, by the processor, a location of a barcode or other product label in the image; and 
 determining, by the processor, an orientation of the product in the image based on the identified location of the barcode or other product label in the image; and 
 
 stitching, by the processor, each image of the second image data associated with the identified product to the three-dimensional mesh associated with the identified product based on the orientation of the product in each image of the second image data associated with the identified product. 
 
     
     
       6. A system for three-dimensional product reconstruction using multiple images collected at POS workstations, comprising:
 a barcode reader associated with a point of sale (POS) workstation configured to capture first image data associated with a plurality of products passing through a product scanning region of the POS workstation and analyze barcode data from the first image data to identify at least some of the plurality of products passing through the product scanning region of the POS workstation; 
 one or more color cameras associated with the POS workstation configured to, responsive to the barcode reader identifying a product, from the plurality of products, resulting in an identified product, capture second image data associated with the identified product; 
 a processor; and 
 a memory storing non-transitory, computer-readable instructions, that, when executed by the processor, cause the processor to generate a textured three-dimensional mesh reconstruction for the identified product based on the second image data associated with the identified product, 
 wherein the instructions, that cause the processor to generate the textured three-dimensional mesh reconstruction associated with the identified product based on the second image data associated with the identified product, include instructions that cause the processor to:
 for each image of the second image data associated with the identified product; 
 segment the identified product from the background, or any hands in the image; 
 analyze the segmented image of the identified product to determine a depth and a texture associated with each pixel of the identified product in the image; and 
 generate a three-dimensional point cloud associated with the segmented image of the identified product based on the depth and the texture associated with each pixel of the identified product; and 
 
 stitch the three-dimensional point clouds associated with each image of the second image data associated with the identified product together to generate a textured three-dimensional mesh reconstruction associated with the identified product, and 
 wherein the instructions, that cause the processor to stitch the three-dimensional point cloud associated with each image of the second image data associated with the identified product together to generate a textured three-dimensional mesh reconstruction associated with the product, include instructions that cause the processor to:
 for each three-dimensional point cloud associated with the identified product:
 identify a location of a barcode or other product label in the three-dimensional point cloud; and 
 determine an orientation of the product in the three-dimensional point cloud based on the identified location of the barcode or other product label in the three-dimensional point cloud; and 
 
 
 stitch the three-dimensional point clouds associated with each image of the second image data associated with the identified product together to generate the textured three-dimensional mesh reconstruction associated with the product, based on the determined orientation of the product in each three-dimensional point cloud. 
 
     
     
       7. The method of  claim 6 , wherein the instructions, that cause the processor to generate the textured three-dimensional mesh reconstruction for the identified product based on the second image data associated with the identified product, include instructions that cause the processor to:
 determine an indication of a shape associated with the identified product; 
 generate a three-dimensional mesh associated with the identified product based on the indication of the shape associated with the identified product; and 
 stitch the second image data associated with the identified product to the three-dimensional mesh associated with the identified product in order to generate a textured three-dimensional mesh associated with the identified product. 
 
     
     
       8. The method of  claim 7 , wherein the instructions, that cause the processor to determine the indication of the shape associated with the identified product, include instructions that cause the processor to access a database storing an indication of a product shape associated with the identified product. 
     
     
       9. The method of  claim 7 , wherein the instructions, that cause the processor to determine the indication of the shape associated with the product shown in the image, include instructions causing the processor to:
 for at least one image of the second image data associated with the identified product:
 segment the identified product from the background, or any hands in the image; and 
 analyze the segmented image of the identified product using a neural network to classify the shape associated with the identified product as one of a plurality of possible product shapes. 
 
 
     
     
       10. The method of  claim 7 , wherein the instructions, that cause the processor to stitch the second image data associated with the identified product to the three-dimensional mesh associated with the identified product in order to generate the textured three-dimensional mesh associated with the identified product, include instructions that cause the processor to:
 for each image of the second image data associated with the identified product:
 segment the identified product from the background, or any hands in the image; 
 identify a location of a barcode or other product label in the image; and 
 determine an orientation of the product in the image based on the identified location of the barcode or other product label in the image; and 
 
 stitch each image of the second image data associated with the identified product to the three-dimensional mesh associated with the identified product based on the orientation of the product in each image of the second image data associated with the identified product.