Image-based kitchen tracking system with order accuracy management

Abstract

The subject matter of this specification can be implemented in, among other things, methods, systems, computer-readable storage medium. A method can include receiving, by a processing device, image data including one or more image frames indicative of a current state of a meal preparation area. The processing device determines one of a meal preparation item or a meal preparation action associated with the current state of the kitchen based on the image data. The processing device receives order data comprising one or more pending meal orders. The processing device can determine an order preparation error based on the order data and at least one of the meal preparation item or the meal preparation action. The processing device causes the order preparation error to be displayed on a graphical user interface (GUI).

1 . (canceled) 
     
     
         2 . A non-transitory machine-readable storage medium storing instructions which, when executed, cause a processing device to perform operations comprising:
 obtaining image data indicative of a state of an order preparation area;   determining an order preparation action performed in the order preparation area based on the image data;   obtaining order data comprising a first pending order;   determining a first association between the order preparation action and the first pending order;   determining that the order preparation action comprises an order preparation error based on the first association; and   causing a device proximate the order preparation area to present a user alert based on the order preparation error.   
     
     
         3 . The non-transitory machine-readable storage medium of  claim 2 , wherein the order preparation action comprises adding an order item to a first order assembly location associated with the first pending order. 
     
     
         4 . The non-transitory machine-readable storage medium of  claim 3 , wherein the order preparation error comprises a discrepancy between the order item and the first pending order. 
     
     
         5 . The non-transitory machine-readable storage medium of  claim 2 , wherein the order preparation action comprises preparing a first order for delivery, the first order corresponding to the first pending order. 
     
     
         6 . The non-transitory machine-readable storage medium of  claim 5 , wherein the order preparation error comprises one or more items of the first pending order being omitted from the first order. 
     
     
         7 . The non-transitory machine-readable storage medium of  claim 2 , wherein the order preparation action comprises packaging an order item, and wherein the order preparation error comprises incorrect packaging of the order item. 
     
     
         8 . The non-transitory machine-readable storage medium of  claim 2 , wherein presenting the user alert comprises one or more of:
 causing an auditory alert;   illuminating a light source;   providing haptic feedback; or   displaying an alert on a user interface (UI).   
     
     
         9 . The non-transitory machine-readable storage medium of  claim 2 , wherein determining the order preparation action comprises:
 providing the image data as input to a trained machine learning model; and   obtaining one or more outputs of the trained machine learning model, the one or more outputs identifying the order preparation action.   
     
     
         10 . The non-transitory machine-readable storage medium of  claim 9 , further comprising:
 receiving a user input indicative of an accuracy of the order preparation error; and   using the user input to update the trained machine learning model.   
     
     
         11 . A non-transitory machine-readable storage medium storing instructions which, when executed, cause a processing device to perform operations comprising:
 obtaining image data comprising one or more image frames indicative of a state of a food item preparation area;   obtaining a food item preparation action associated with the state of the food item preparation area based on the image data;   obtaining an indication of steps associated with preparing a target food item;   determining that the food item preparation action is associated with the target food item;   determining a food item preparation error based on the steps associated with preparing the target food item and the food item preparation action; and   
       causing a device proximate the food item preparation area to present a user alert based on the food item preparation error. 
     
     
         12 . The non-transitory machine-readable storage medium of  claim 11 , wherein determining that the food item preparation action is associated with the target food item comprises:
 determining, based on the image data, a plurality of food item preparation actions performed in connection with a food item in preparation, wherein the food item preparation action is to be performed in connection with the food item in preparation; and   determining, based on a library of food items, that the plurality of food item preparation actions correspond to the target food item.   
     
     
         13 . The non-transitory machine-readable storage medium of  claim 11 , wherein determining that the food item preparation action is associated with the target food item comprises:
 determining, based on the image data, a plurality of food items;   determining that the plurality of food items are associated with a pending food order; and   determining that a food item in preparation is associated with the pending food order, wherein the pending food order comprises the target food item, and wherein the food item in preparation is associated with the food item preparation action.   
     
     
         14 . The non-transitory machine-readable storage medium of  claim 11 , wherein determining that the food item preparation action is associated with the target food item comprises:
 obtaining, by the processing device, a pending food order comprising the target food item; and   determining, by the processing device, that a food item in preparation is associated with the pending food order, wherein the food item in preparation is associated with the food item preparation action.   
     
     
         15 . The non-transitory machine-readable storage medium of  claim 14 , wherein the pending food order is obtained from a point-of-sale system. 
     
     
         16 . The non-transitory machine-readable storage medium of  claim 11 , wherein the food item preparation error comprises performance of a food preparation step that is not associated with the target food item. 
     
     
         17 . The non-transitory machine-readable storage medium of  claim 11 , wherein the food item preparation error comprises providing a quantity of an ingredient for a food item in preparation that is outside a threshold range of quantities associated with a first step for preparing the target food item of the steps associated with preparing the target food item. 
     
     
         18 . A system, comprising memory and a processing device coupled to the memory, wherein the processing device is configured to:
 obtain image data indicative of a state of an order preparation area;   identify a food item associated with the order preparation area;   obtain order data comprising a pending order;   determine that the food item is associated with the pending order;   determine an order preparation error in connection with the food item and the pending order; and   cause a device proximate the order preparation area to present a user alert based on the order preparation error.   
     
     
         19 . The system of  claim 18 , wherein determining that the food item is associated with the pending order comprises determining that the food item is proximate one or more order items, and wherein the pending order does not comprise the food item. 
     
     
         20 . The system of  claim 18 , wherein determining that the food item is associated with the pending order comprises identifying the food item and determining that the pending order comprises the food item, and wherein the user alert comprises an indication that the food item is not proximate to one or more items associated with the pending order. 
     
     
         21 . The system of  claim 18 , wherein an indication that the food item is not proximate to one or more items associated with the pending order comprises an indication that the food item is not packaged in a food packaging system associated with the pending order.