Method for automatically generating planograms of shelving structures within a store

Abstract

One variation of a method for automatically generating a planogram for a store includes: dispatching a robotic system to autonomously navigate within the store during a mapping routine; accessing a floor map of the floor space generated by the robotic system from map data collected during the mapping routine; identifying a shelving structure within the map of the floor space; defining a first set of waypoints along an aisle facing the shelving structure; dispatching the robotic system to navigate to and to capture optical data at the set of waypoints during an imaging routine; receiving a set of images generated from optical data recorded by the robotic system during the imaging routine; identifying products and positions of products in the set of images; and generating a planogram of the shelving segment based on products and positions of products identified in the set of images.

I claim: 
     
         1 . A method for automatically generating a planogram assigning products to shelving structures within a store, the method comprising:
 dispatching a robotic system to image a set of shelving structures within the store during a first scan cycle;   accessing a set of images captured by the robotic system during the first scan cycle; and   based on confirmation of a preferred stocking condition in the store during the first scan cycle:
 initializing a new planogram of the store; 
 detecting a set of products depicted in the set of images; and 
 for each product in the set of products detected in the set of images:
 identifying the product; 
 calculating a location of the product in the store based on:
 a location of the robotic system in the store during capture of a particular image, in the set of images, depicting the product; and 
 a position of the product depicted in the particular image; and 
 
 writing a product assignment, linking a product identifier of the product to the location of the product, to the new planogram. 
 
   
     
     
         2 . The method of  claim 1 :
 further comprising accessing a spatial map of the store;   wherein detecting the set of products depicted in the set of images comprises detecting a first product, in the set of products, depicted in a first image in the set of images; and   wherein calculating a location of a product in the store for each product in the set of products comprises, for the first product in the set of products:
 calculating a first field of view of a camera on the robotic system that captured the first image; 
 projecting the first field of view of the camera onto the spatial map of the store based on a location and an orientation of the robotic system in the store during capture of the first image by the camera; 
 projecting a first position of the first product, extracted from the first image, into the first field of view of the camera projected onto the spatial map of the store; and 
 calculating a first location of the first product based on the first position of the product projected into the first field of view of the camera projected onto the spatial map of the store. 
   
     
     
         3 . The method of  claim 1 :
 further comprising accessing a spatial map of the store;   wherein initializing the new planogram comprises initializing a first elevation map of a first shelving structure, in the set of shelving structures, in the new planogram based on a first location and a first geometry of the first shelving structure represented in the spatial map;   wherein calculating a location of a product in the store for each product in the set of products comprises, for a first product in the set of products:
 calculating a first shelf position of the first product based on a first vertical position of the first product detected in a first image, in the set of images, depicting the first shelving structure; and 
 calculating a first lateral position of the first product based on a first horizontal position of the first product detected in the first image depicting the first shelving structure; and 
   wherein writing a product assignment to the new planogram for each product in the set of products comprises, for the first product:
 locating a first product assignment, specifying the first product, on the first elevation map of the first shelving structure according to the first shelf position of the first product and the first lateral position of the first product. 
   
     
     
         4 . The method of  claim 1 :
 wherein detecting the set of products depicted in the set of images and calculating a location of a product in the store for each product in the set of products comprises, for a first product in the set of products:
 detecting a first shelf at a first vertical position in a first shelving structure depicted in a first image in the set of images; 
 detecting a first object within a first region of the first image corresponding to a first lateral position over the first shelf; and 
 identifying the first object as a first unit of the first product based on features extracted from the first region of the first image; and 
   wherein writing a product assignment to the new planogram for each product in the set of products comprises, for the first product:
 projecting the first vertical position of the first shelf and the first lateral position of the first object onto a representation of the first shelving structure in the new planogram to define a first slot in the new planogram; and 
 writing a first product identifier of the first product to the first slot in the new planogram. 
   
     
     
         5 . The method of  claim 4 , wherein identifying the first object as the first unit of the first product comprises:
 detecting a first textual string in the first region of the first image;   identifying a first brand associated with the first object based on the first textual string;   retrieving a first set of template images from a database of template images, each template image in the first set of template images representing a product associated with the first brand; and   identifying the first object as the first unit of the first product in response to detecting features, in the first region of the first image, analogous to features in a first template image, in the first set of template images, corresponding to the first product.   
     
     
         6 . The method of  claim 1 , wherein dispatching the robotic system to image the set of shelving structures within the store during the first scan cycle comprises dispatching the robotic system to execute the first scan cycle within the store following initial delivery of the robotic system to the store and in response to absence of a preexisting planogram of the store. 
     
     
         7 . The method of  claim 1 , wherein dispatching the robotic system to image the set of shelving structures within the store during the first scan cycle comprises dispatching the robotic system to autonomously navigate toward and to image the set of shelving structures within the store during the first scan cycle in response to receipt of confirmation of the preferred stocking condition in the store. 
     
     
         8 . The method of  claim 1 :
 further comprising:
 accessing a second set of images captured by the robotic system during a second scan cycle preceding the first scan cycle; 
 detecting a second set of products in the second set of images; 
 interpreting a second stock condition in the store during the second scan cycle based on the second set of products; 
 detecting a set of differences between the second stock condition and a second planogram of the store current to the second scan cycle; 
 generating a second global restocking list specifying restocking of a subset of slots in the set of shelving structures in the store based on the set of differences; and 
 serving the second global restocking to computing devices affiliated with associates of the store during a scheduled global restocking period succeeding the second scan cycle; 
   wherein dispatching the robotic system to image the set of shelving structures within the store during the first scan cycle comprises dispatching the robotic system to autonomously navigate toward and to image the set of shelving structures within the store during the first scan cycle succeeding the second scan cycle and the scheduled global restocking period; and   further comprising, in response to confirmation of the preferred stocking condition in the store during the first scan cycle, replacing the second planogram with the new planogram.   
     
     
         9 . The method of  claim 8 :
 wherein detecting the set of products depicted in the set of images comprises detecting a first object depicted in a first region of a first image in the set of images;   wherein identifying the product for each product in the set of products detected in the set of images comprises:
 accessing a list of products assigned to a first shelving structure, depicted in the first image, by the second planogram; 
 accessing a set of template images associated with products in the list of products; 
 extracting a first set of features from the first region of the first image; and 
 identifying the first object as a first product based on alignment between features in the first set of features and features in a first template image in the set of template images, the first template image representing the first object; and 
   wherein replacing the second planogram with the new planogram comprises replacing the second planogram with the new planogram to reflect shifts in locations of products, in the set of products, on shelves in the set of shelving structures throughout the store from the second scan cycle to the first scan cycle.   
     
     
         10 . The method of  claim 1 :
 wherein detecting the set of products depicted in the set of images comprises detecting a first product, in the set of products, depicted in a first region of a first image in the set of images;   wherein calculating a location of a product in the store for each product in the set of products comprises, for the first product in the set of products:
 detecting a set of shelf tags in the first image; 
 associating the first product with a first shelf tag, in the set of shelf tags, depicted proximal the first region of the first image; and 
 defining a first location of the first product in the store relative to the first shelf tag. 
   
     
     
         11 . The method of  claim 10 , further comprising:
 extracting a first distance from the first product to the first shelf tag depicted in the first image;   in response to the first distance exceeding a threshold distance, generating a prompt to adjust the first shelf tag; and   serving the prompt to a computing device affiliated with the store.   
     
     
         12 . The method of  claim 10 :
 wherein identifying a product for each product in the set of products detected in the set of images comprises determining a first product identifier of the first product based on features extracted from the first region of the first image; and   further comprising:
 reading a target product identifier from the first shelf tag depicted in the first image; and 
 in response to detecting a difference between the first product identifier and the target product identifier:
 generating a prompt to reconcile the first product and the first shelf tag; and 
 serving the prompt to a computing device affiliated with the store. 
 
   
     
     
         13 . The method of  claim 1 :
 wherein calculating a location of a product in the store for each product in the set of products comprises, for a first product in the set of products, detecting the first product in a first slot in a first shelving structure in the set of shelving structures;   wherein writing a product assignment to the new planogram for each product in the set of products comprises, for a first product in the set of products, writing a first product assignment, linking a first product identifier of the first product to the first slot in the first shelving structure, to the new planogram; and   further comprising:
 dispatching the robotic system to execute a second scan cycle succeeding the first scan cycle; 
 accessing a second image of the first shelving structure captured by the robotic system during the second scan cycle; 
 identifying a second region in the second image depicting the first slot; 
 based on the first product assignment in the new planogram linking the first product identifier of the first product to the first slot in the first shelving structure, accessing a set of template images associated with the first product identifier; 
 identifying improper stocking of the first product in the first slot during the second scan cycle in response to absence of features, in the second region of the second image, analogous to features in template images in the set of template images; and 
 in response to identifying improper stocking of the first product in the first slot, generating a first restocking prompt for the first slot. 
   
     
     
         14 . The method of  claim 1 :
 further comprising:
 prior to the first scan cycle, dispatching the robotic system to autonomously navigate throughout the store during a mapping cycle; 
 accessing a spatial map of the store generated from spatial data captured by the robotic system during the mapping cycle; 
 defining a coordinate system within the spatial map; 
 identifying a first shelving structure represented in the spatial map; and 
 defining a first set of waypoints, relative to the coordinate system, along the first shelving structure, the first set of waypoints specifying orientations facing the first shelving structure; and 
   wherein dispatching the robotic system to image the set of shelving structures within the store during the first scan cycle comprises dispatching the robotic system to navigate to each waypoint in the first set of waypoints and to record an image of the first shelving structure while occupying each waypoint in the first set of waypoints.   
     
     
         15 . A method for automatically generating a planogram assigning products to shelving structures within a store, the method comprising:
 dispatching a robotic system to image a set of shelving structures within the store during a first scan cycle;   accessing a set of images captured by the robotic system during the first scan cycle;   initializing a new planogram of the store;   detecting a first set of shelf tags depicted in the set of images; and   for each shelf tag in the first set of shelf tags detected in the set of images:
 reading a product identifier from the shelf tag; 
 calculating a location of the shelf tag in the store based on:
 a location of the robotic system in the store during capture of a particular image, in the set of images, depicting the shelf tag; and 
 a position of the shelf tag depicted in the particular image; and 
 
 writing a product assignment, linking the product identifier to the location of the shelf tag, to the new planogram. 
   
     
     
         16 . The method of  claim 15 , wherein initializing the new planogram of the store comprises initializing the new planogram of the store in response to receipt of confirmation of a preferred stocking condition in the store during the first scan cycle. 
     
     
         17 . The method of  claim 15 :
 wherein reading a product identifier from a shelf tag and writing a product assignment to the new planogram for each shelf tag in the first set of shelf tags comprises, for each shelf tag in the first set of shelf tags:
 reading a product identifier from the shelf tag with a confidence score; and 
 writing a product assignment, generated based on the shelf tag, to the new planogram in response to the confidence score exceeding a threshold confidence; and 
   further comprising:
 detecting a second set of shelf tags depicted in the set of images; and 
 for each shelf tag in the second set of shelf tags:
 reading a product identifier from the shelf tag with a confidence score; and 
 in response to the confidence score falling below the threshold confidence:
 detecting a product proximal the shelf tag in an image, in the set of images, depicting the shelf tag; 
 identifying the product based on features extracted from a region of the image depicting the product; and 
 writing a product assignment, linking a product identifier of the product to the location of the shelf tag, to the new planogram. 
 
 
   
     
     
         18 . The method of  claim 15 , wherein reading a product identifier from a shelf tag and writing a product assignment to the new planogram for each shelf tag in the first set of shelf tags comprises, for a first shelf tag in the first set of shelf tags:
 reading a first product identifier from the first shelf tag;   detecting a first product proximal the first shelf tag in a first image, in the set of images, depicting the first shelf tag;   identifying the first product based on features extracted from a first region of the first image depicting the first product; and   writing a first product assignment, linking the first product identifier to a first location of the first shelf tag, to the new planogram in response to correspondence between the first product identifier and the first product.   
     
     
         19 . The method of  claim 18 , further comprising:
 detecting a second shelf tag in a second image in the set of images;   reading a second product identifier from the second shelf tag;   detecting a second product proximal the second shelf tag in the second image;   identifying the second product based on features extracted from a second region of the image depicting the second product; and   in response to detecting a difference between the second product identifier and the second product:
 generating a prompt to reconcile the second product and the second shelf tag; and 
 serving the prompt to a computing device affiliated with the store. 
   
     
     
         20 . A method for automatically generating a planogram assigning products to shelving structures within a store, the method comprising:
 dispatching a robotic system to image a set of shelving structures within the store during a first scan cycle;   accessing a set of images captured by the robotic system during the first scan cycle;   detecting a set of products depicted in the set of images;   for each product in the set of products detected in the set of images:
 identifying the product; and 
 calculating a location of the product in the store based on a position of the product depicted in the particular image; and 
   based on confirmation of a preferred stocking condition in the store during the first scan cycle:
 initializing a new planogram of the store; and 
 for each product in the set of products detected in the set of images, writing a product assignment, linking a product identifier of the product to a location of the product, to the new planogram.