Identifying products from on-shelf sensor data and visual data

Abstract

A non-transitory computer-readable medium includes instructions that when executed by a processor cause the processor to perform a method for identifying products from on-shelf sensors and image data. The method may include receiving data captured using a plurality of sensors positioned between at least part of a retail shelf and one or more products placed on the at least part of the retail shelf. The method may also include receiving an image of the at least part of the retail shelf and at least one of the one or more products. The method may also include analyzing the captured data and the image to determine a product type of the one or more products.

What is claimed is: 
     
         1 . A non-transitory computer-readable medium including instructions that when executed by a processor cause the processor to perform a method for identifying products from on-shelf sensors and image data, the method comprising:
 receiving data captured using a plurality of sensors positioned between at least part of a retail shelf and one or more products placed on the at least part of the retail shelf;   receiving an image of the at least part of the retail shelf and at least one of the one or more products; and   analyzing the captured data and the image to determine a product type of the one or more products.   
     
     
         2 . The non-transitory computer-readable medium of  claim 1 , wherein the one or more products include at least one product not depicted in the image. 
     
     
         3 . The non-transitory computer-readable medium of  claim 1 , wherein the captured data is indicative of a weight of a product among the one or more products. 
     
     
         4 . The non-transitory computer-readable medium of  claim 1 , wherein the captured data is indicative of a pressure caused by a product among the one or more products. 
     
     
         5 . The non-transitory computer-readable medium of  claim 1 , wherein the captured data is indicative of a footprint of a product among the one or more products. 
     
     
         6 . The non-transitory computer-readable medium of  claim 1 , wherein the captured data is indicative of ambient light captured by at least one of the plurality of sensors. 
     
     
         7 . The non-transitory computer-readable medium of  claim 1 , wherein the image includes depth information of the at least one of the one or more products. 
     
     
         8 . The non-transitory computer-readable medium of  claim 1 , wherein the method further comprises analyzing the captured data and the image to determine a quantity of the one or more products. 
     
     
         9 . The non-transitory computer-readable medium of  claim 1 , wherein the method further comprises analyzing the captured data and the image to determine facings of the one or more products. 
     
     
         10 . The non-transitory computer-readable medium of  claim 1 , wherein the method further comprises analyzing the captured data and the image to determine quality of the one or more products. 
     
     
         11 . The non-transitory computer-readable medium of  claim 1 , wherein the method further comprises:
 analyzing the captured data and the image to determine an action associated with the retail shelf; and   providing information configured to cause the performance of the action.   
     
     
         12 . The non-transitory computer-readable medium of  claim 1 , wherein the method further comprises:
 analyzing the captured data to determine a need to capture visual data; and in response to the determined need to capture visual data, triggering the capture of the image.   
     
     
         13 . The non-transitory computer-readable medium of  claim 1 , wherein the method further comprises:
 analyzing the captured data to determine a plurality of alternative candidate product types; and   analyzing the image to select the product type of the one or more products from the plurality of alternative candidate product types.   
     
     
         14 . The non-transitory computer-readable medium of  claim 1 , wherein the method further comprises:
 analyzing the image to determine a plurality of alternative candidate product types; and   analyzing the captured data to select the product type of the one or more products from the plurality of alternative candidate product types.   
     
     
         15 . The non-transitory computer-readable medium of  claim 1 , wherein analyzing the captured data and the image to determine the product type of the one or more products comprises:
 analyzing the image to determine a first product type associated with the one or more products;   analyzing the captured data to determine a second product type associated with the one or more products; and   determining the product type of the one or more products based on a comparison of the first and second product types.   
     
     
         16 . The non-transitory computer-readable medium of  claim 1 , wherein analyzing the captured data and the image to determine the product type of the one or more products comprises:
 extracting one or more features from the captured data and one or more features from the image;   analyzing the one or more extracted features from the captured data and the one or more features from the image using a product recognition model; and   determining the product type of the one or more products based on an output of the product recognition model.   
     
     
         17 . The non-transitory computer-readable medium of  claim 1 , wherein analyzing the captured data and the image to determine the product type of the one or more products comprises:
 calculating at least one convolution of at least part of the image; and   analyzing the calculated at least one convolution of at least part of the image and the captured data to determine the product type of the one or more products.   
     
     
         18 . The non-transitory computer-readable medium of  claim 1 , wherein the captured data includes at least an array of values, and wherein analyzing the captured data and the image to determine the product type of the one or more products comprises:
 calculating at least one convolution of at least part of the image;   calculating at least one convolution of at least part of the array of values; and   analyzing the calculated at least one convolution of at least part of the image and the at least one convolution of the array of values to determine the product type of the one or more products.   
     
     
         19 . A method for identifying products from on-shelf sensors and image data, the method comprising:
 receiving data captured using a plurality of sensors positioned between at least part of a retail shelf and one or more products placed on the at least part of the retail shelf;   receiving an image of the at least part of the retail shelf and at least one of the one or more products; and   analyzing the captured data and the image to determine a product type of the one or more products.   
     
     
         20 . A system for identifying products from on-shelf sensors and image data, the system comprising:
 at least one processor programmed to:
 receive data captured using a plurality of sensors positioned between at least part of a retail shelf and one or more products placed on the at least part of the retail shelf; 
 receive an image of the at least part of the retail shelf and at least one of the one or more products; and 
 analyze the captured data and the image to determine a product type of the one or more products.