Recognition and prediction of semantic events learned through repeated observation

Abstract

A system for managing consumer packaged goods (CPGs) is disclosed. The system includes a front end and a back end. The front end includes a mobile, airborne platform equipped with a digital image capturing device, and a wireless communications device. The backend is in communication with the front end via the wireless communications device, and includes a combinational convolutional neural network which derives models based on input data, a convolutional neural network which generates perception scoring utilizing input put from said combinational convolutional neural network, and a recurrent neural network which makes behavior predictions based on input from said convolutional neural network. The front end captures images of CPGs on a shelf, wherein the CPGs are subject to depletion over time, and wherein the backend generates predictions regarding the depletion state of the CPGs on the shelf as a function of time.

What is claimed is: 
     
         1 . A system for managing consumer packaged goods (CPGs), comprising:
 a front end which includes a mobile, airborne platform equipped with   (a) a digital image capturing device, and   (b) a wireless communications device; and   a backend, in communication with said front end via said wireless communications device, said back end including
 (a) a combinational convolutional neural network which derives models based on input data, 
 (b) a convolutional neural network which generates perception scoring utilizing input put from said combinational convolutional neural network, and 
 (c) a recurrent neural network which makes behavior predictions based on input from said convolutional neural network; 
   
       wherein said front end captures images of CPGs on a shelf, wherein the CPGs are subject to depletion over time, and wherein the backend generates predictions regarding the depletion state of the CPGs on the shelf as a function of time. 
     
     
         2 . The system of  claim 1 , wherein the behavior predictions generated by the recurrent neural network include the state of CPGs on a shelf as a function of time. 
     
     
         3 . The system of  claim 2 , wherein said backend further includes a state machine which ascertains the actual state of CPGs on a shelf at a given time, and wherein said recurrent neural network compares the predicted state of CPGs to the actual state of the CPGs and inputs the results to the combinational convolutional neural network. combinational convolutional neural network 
     
     
         4 . The system of  claim 1 , further comprising:
 a graphical user interface (GUI) which includes a dashboard that displays the status of CPG items on a shelf at a physical store.   
     
     
         5 . The system of  claim 4 , wherein the dashboard displays CPGs that have run out. 
     
     
         6 . The system of  claim 4 , wherein the dashboard displays CPGs that have partially run out. 
     
     
         7 . The system of  claim 4 , wherein the dashboard displays the time at which each CPG in a set of CPGs is predicted to run out. 
     
     
         8 . The system of  claim 7 , wherein the time at which each CPG in a set of CPGs is predicted to run out is generated by said recurrent neural network based on input from said convolutional neural network. 
     
     
         9 . The system of  claim 1 , wherein said recurrent neural network generates imputed planograms which predict a future depletion state that each of the CPGs is being managed to. 
     
     
         10 . The system of  claim 1 , wherein the convolutional neural network develops derived inventory maps of the CPGs on the shelf. 
     
     
         11 . The system of  claim 10 , further comprising:
 a time series database, wherein said database includes arrays of derived inventory maps indexed by time.   
     
     
         12 . The system of  claim 11 , wherein the recurrent neural network operates on said time series database to generate imputed planograms which predict the depletion state that the CPGs are currently being managed to. 
     
     
         13 . The system of  claim 12 , wherein each CPG has a stock keeping unit (SKU) associated with it, and further comprising:
 a convolutional neural network model which specifies the fullness state of each SKU, wherein the fullness state is a ratio having a numerator and a denominator, wherein the denominator is the number of CPGs associated with a particular SKU that are present on the shelf when the shelf is fully stocked, and wherein the numerator is the number of CPGs associated with the particular SKU that are currently on the shelf.   
     
     
         14 . The system of  claim 13 , further comprising a set of current state conclusions about current state cognitive anomalies. 
     
     
         15 . The system of  claim 14 , wherein said cognitive anomalies are selected from the group of conditions consisting of (a) an item being out of stock, (b) an item being misplaced, or (c) the width of a row of CPGs is different than the width in the imputed planogram. 
     
     
         16 . The system of  claim 15 , further comprising:
 a time series analysis module which conducts a time series analysis on the current state cognitive anomalies to predict future state anomalies.   
     
     
         17 . The system of  claim 12 , wherein each CPG has a stock keeping unit (SKU) associated with it, and further comprising:
 a convolutional neural network model which specifies the velocity of change for CPGs associated with each SKU.   
     
     
         18 . The system of  claim 17 , wherein the convolutional neural network uses the convolutional neural network model to predict the future state of CPGs associated with each SKU. 
     
     
         19 . A method for managing an inventory of consumer packaged goods (CPGs), comprising:
 (A) using a convolutional neural network, in conjunction with object recognition of CPGs and physical inventory localization, to generate a plurality of derived inventory maps of CPGs over a corresponding plurality of points of time;   (B) using a first recurrent neural network (RNN) to generate an imputed planogram by performing time series analysis on the plurality of derived inventory maps;   (C) using a convolutional neural network (CNN) to derive a fullness of stock score for the inventory of CPGs relative to the imputed planogram;   (D) using the fullness of stock score, in conjunction with the imputed planogram, to obtain a current state anomaly classification;   repeating steps A-D n times, wherein n> 1 ;   using a second recurrent neural network (RNN) to perform time series analysis on changes to current state anomalies as a function of time, thereby obtaining time series analysis results; and   using the time series analysis results to predict at least one future state anomaly classification.   
     
     
         20 . The method of  claim 19 , wherein said current state anomaly classification and said future state anomaly classification relate to the state of a CPG and are selected from the group consisting of the state of being out-of-stock, the state of being partially stocked, the state of being misplaced, and the state of being mis-faced. 
     
     
         21 . The method of  claim 19 , wherein deriving a fullness of stock score for the inventory of CPGs relative to the imputed planogram includes utilizing a fractional gradation based on the fraction n/m, where n, m∈Z, and whereby each CPG is assigned a fullness of stock count selected from the group consisting of 0, n/m, 2 n/m, . . . , (m−1)/m, and 1, wherein 0 represents the state of being out-of-stock, wherein 1 represents the state of being fully stocked, and wherein k/m represents the state of being k/m fully stocked for k∈[1, . . . , m]. 
     
     
         22 . The method of  claim 19 , wherein deriving a fullness of stock score for the inventory of CPGs relative to the imputed planogram includes utilizing a quarter gradation, whereby each CPG is assigned a fullness of stock count selected from the group consisting of empty, ¼ full, ½ full, ¾ full, and full. 
     
     
         23 . The method of  claim 19 , wherein using the time series analysis results to predict at least one future state anomaly classification includes generating a planogram which depicts the predicted state of a plurality of CPGs at a future point in time. 
     
     
         24 . The method of  claim 19 , wherein using the time series analysis results to predict at least one future state anomaly classification includes generating n planograms, wherein n≥2, and wherein each of said n planograms depicts the predicted state of each of a plurality of CPGs at one of n distinct future points in time. 
     
     
         25 . A method for managing an inventory of consumer packaged goods (CPGs), comprising:
 (A) obtaining object recognition data by
 (a) identifying a set of objects as being a set of CPGs by applying a first level of object recognition to the set of objects, 
 (b) performing a second level of object recognition on each of the objects, 
 (c) assigning each of the objects to one of a plurality of predefined superclasses, based on the results of the second level of object recognition, 
 (d) obtaining cropped images of each of the objects, 
 (e) performing a third level of object recognition on the cropped images, and 
 (f) assigning each object to one of a plurality of predefined subclasses, based on the results of the third level of object recognition; 
   (B) using a convolutional neural network, in conjunction with the object recognition data and physical inventory localization, to generate a plurality of derived inventory maps of CPGs over a corresponding plurality of points of time;   (C) using a first recurrent neural network (RNN) to generate an imputed planogram by performing time series analysis on the plurality of derived inventory maps;   (D) using a convolutional neural network (CNN) to derive a fullness of stock score for the inventory of CPGs relative to the imputed planogram;   (E) using the fullness of stock score, in conjunction with the imputed planogram, to obtain a current state anomaly classification;   repeating steps B-E n times, wherein n>1;   using a second recurrent neural network (RNN) to perform time series analysis on changes to current state anomalies as a function of time, thereby obtaining time series analysis results; and   using the time series analysis results to predict at least one future state anomaly classification.   
     
     
         26 . The method of  claim 25 , wherein said current state anomaly classification and said future state anomaly classification relate to the state of a CPG and are selected from the group consisting of the state of being out-of-stock, the state of being partially stocked, the state of being misplaced, and the state of being mis-faced. 
     
     
         27 . The method of  claim 25 , wherein deriving a fullness of stock score for the inventory of CPGs relative to the imputed planogram includes utilizing a fractional gradation based on the fraction n/m, where n, m∈Z, and whereby each CPG is assigned a fullness of stock count selected from the group consisting of 0, n/m, 2 n/m, . . . , (m−1)/m, and 1, wherein 0 represents the state of being out-of-stock, wherein  1  represents the state of being fully stocked, and wherein k/m represents the state of being k/m fully stocked for k∈[1, . . . , m]. 
     
     
         28 . The method of  claim 25 , wherein deriving a fullness of stock score for the inventory of CPGs relative to the imputed planogram includes utilizing a quarter gradation, whereby each CPG is assigned a fullness of stock count selected from the group consisting of empty, ¼ full, ½ full, ¾ full, and full. 
     
     
         29 . The method of  claim 25 , wherein using the time series analysis results to predict at least one future state anomaly classification includes generating a planogram which depicts the predicted state of a plurality of CPGs at a future point in time. 
     
     
         30 . The method of  claim 25 , wherein using the time series analysis results to predict at least one future state anomaly classification includes generating n planograms, wherein n≥2, and wherein each of said n planograms depicts the predicted state of each of a plurality of CPGs at one of n distinct future points in time.