Method and apparatus for providing virtual touch interaction in the drive-thru

Abstract

The present invention is a method and apparatus for providing an enhanced automatic drive-thru experience to the customers in a vehicle by allowing use of natural hand gestures to interact with digital content. The invention is named Virtual Touch Ordering System (VTOS). In the VTOS, the virtual touch interaction is defined to be a contact free interaction, in which a user is able to select graphical objects within the digital contents on a display system and is able to control the processes connected to the graphical objects, by natural hand gestures without touching any physical devices, such as a keyboard or a touch screen. Using the virtual touch interaction of the VTOS, the user is able to complete transactions or ordering, without leaving the car and without any physical contact with the display. A plurality of Computer Vision algorithms in the VTOS processes a plurality of input image sequences from the image-capturing system that is pointed at the customers in a vehicle and performs the virtual touch interaction by natural hand gestures. The invention can increase the throughput of drive-thru interaction and reduce the delay in wait time, labor cost, and maintenance cost.

1. A method for interacting with a service system from inside a vehicle  implementing a transaction without physical contact with the system , the method comprising steps of :
 (a) showing  providing visual information on  to a person through a display,  ;  
 (b) capturing a plurality of input images of a  the person, who wants to use the system,  interacting with the visual information and the  of a vehicle in which said person is sitting,  located; 
 (c) processing said plurality of input images in order  to extract motion information in a contact-free manner,  ;  
 (d) performing a contact- free  interaction based on the extracted motion information; and that allows said person to interact with the shown visual information, and  
 (e) processing the interaction results of said person with said service system, 
 wherein the step of performing  the contact-free interaction is possible  occurs regardless of whether a window of said vehicle's window  is open or closed.  
 
     
     
       2. The method according to  claim 1 , wherein the step of  performing the contact- free interaction comprises: 
 processing the interaction results involves  extracted motion information; and  
 updating said visual information on a  the display based at least in part on the processing of the extracted motion information, wherein the visual information is updated when said person is looking at the display system .  
 
     
     
       3. The method according to  claim 1 , wherein the step of showing  visual information on a display further comprises a step for showing elements that can be selected or manipulated by the contact-free interaction,
 whereby said visual information can be shown on an electronic display or on a static display board,  
 whereby said visual information shows products or  includes at least one of a menu or food items for purchase. , and  
 whereby exemplary embodiments of said elements can be menus, graphical depictions of buttons, and icons.  
 
     
     
       4. The method according to  claim 1 , wherein the step of capturing said  plurality of input images of the person and the contact-free interaction further comprises a step for  are captured using one or a plurality of image-capturing devices. for the capturing. 
     
     
       5. The method according to  claim 1 , wherein the contact-free interaction further comprises a step for allowing a certain degree of spatial freedom in parking said vehicle and interacting with the system by said person,
 whereby the spatial freedom can be realized  plurality of input images are captured in an exemplary maximum interaction range volume and an exemplary  an optimal interaction volume in a virtual space, which change according to the  wherein a position of the optimal interaction volume is based at least in part on a physical dimension of the person in the vehicle  and a physical dimension of the vehicle.  
 
     
     
       6. The method according to  claim 1 , wherein the method  further comprises a step for  ing adjusting the main content display region in the display system according to the vehicle's parking position and said person's height,
 whereby the adjustment makes said person easier to interact with the system even if various vehicle have different heights depending on the type of the vehicle, and  
 whereby the other parts of the display screen, where the main content is not shown, can be used for advertisement or promotion.  a position of the visual information on the display based at least in part on a position of the vehicle and at least in part on a position of the person in the vehicle. 
 
     
     
       7. The method according to  claim 1 , wherein the step of processing the interaction results further comprises a step for  further comprising collecting a plurality of  data about at least one of said person and  or the contact- free  interaction,
 whereby exemplary statistics gathered by the data collection can include, thewherein the collected data includes at least one of an amount of time to finish the contact- free  interaction, the division of people ina demographic groupsof the person, the traffic measurement, such as  traffic composition by time of day, day of week, and demographic shifts, and theor customer behavior. , such as the time spent at a particular item selection screen or whether purchases are made or not. 
 
     
     
       8. The method according to  claim 1 , wherein the step of processing said plurality of input images in order to extract motion information in a contact-free manner further comprises a step for processing the detection of said person's vehicle and a localization of the vehicle  further comprising identifying a location of the window of the vehicle. 
     
     
       9. The method according to  claim 1 , wherein the step of processing said  plurality of input images in order to extract motion information in a contact-free manner further comprises a step for performing  are captured using face detection and hand tracking. 
     
     
       10. The method according to  claim 1 , wherein the step of performing interaction further comprises a step for allowing said person to purchase food items or non-food items, such as pharmaceuticals, or where it is designed to provide services, such as banking, using the contact-free interface.  contact- free interaction comprises at least one of a food order, a pharmaceutical order, or a banking transaction.   
     
     
       11. An apparatus for interacting with a service system from inside a vehicle without physical contact with the system  implementing a transaction without physical contact, comprising:
 (a) means for showing  providing visual information,  to a person in a vehicle; 
 (b) means for capturing a plurality of input images of a  the person, who wants to use the system,  interacting with the visual information and of the vehicle; in which said person is sitting,  
 (c) means for processing said plurality of input images in order  to extract motion information in a contact-free manner,  
 (d) means for performing a contact- free  interaction based on the extracted motion information; and that allows said person to interact with the shown visual information, and  
 (e) means for processing the interaction results of said person with said service system, 
 wherein the means for performing  contact-free interaction is possible  performed regardless of whether said vehicle's  a window of the vehicle is open or closed.  
 
     
     
       12. The apparatus according to  claim 11 , wherein the means for  further comprising processing the interaction results involves  extracted motion information and updating said visual information when said person is looking at the means for showing visual information.  based on the extracted motion information. 
     
     
       13. The apparatus according to  claim 11 , wherein the means for showing  visual information further  comprises means for showing  elements that can  configured to be selected or manipulated by the contact-free interaction,
 whereby said visual information can be shown on means for electronic display or means for static display,  
 whereby said visual information shows products or food items for purchase, and  
 whereby exemplary embodiments of said elements can be menus, graphical depictions of buttons, and icons.  person. 
 
     
     
       14. The apparatus according to  claim 11 , wherein the means for capturing said plurality of input images of the person and the contact-free interaction further comprises means for using one or a plurality of image-capturing devices for the capturing. 
     
     
       15. The apparatus according to  claim 11 , wherein the contact-free interaction further comprises means for allowing a certain degree of spatial freedom in parking said vehicle and interacting with the system by said person,
 whereby the spatial freedom can be realized in an exemplary  visual information is provided to the person within a maximum interaction range volume, and an exemplary optimal interaction volume in a virtual space, which change according to the  wherein the maximum interaction range volume is based at least in part on a physical dimension of the person in the vehicle and the  vehicle.  
 
     
     
       16. The apparatus according to  claim 11 , wherein the apparatus  further comprises  ing means for adjusting a position of the main content display region in the means for showing  visual information according to the vehicle's parking position and said person's height,
 whereby the adjustment makes said person easier to interact with the system even if various vehicle have different heights depending on the type of the vehicle, and  
 whereby the other parts of the display screen, where the main content is not shown, can be used for advertisement or promotion.  based at least in part on a position of the vehicle. 
 
     
     
       17. The apparatus according to  claim 11 , wherein the means for processing the interaction results  further comprises  ing means for collecting a plurality of  data about said  related to at least one of the person and the contact- free  interaction, whereby exemplary statistics gathered by the data collection can include, the  wherein the data includes at least one of an amount of time to finish the contact- free  interaction, the division of people in  a demographic groups  of the person, the  an amount of traffic measurement , such as  a traffic composition, or by time of day, day of week, and demographic shifts, and the  customer behavior. , such as the time spent at a particular item selection screen or whether purchases are made or not. 
     
     
       18. The apparatus according to  claim 11 , wherein the  further comprising means for processing said plurality of input images in order to extract motion information in a contact-free manner further comprises means for processing the detection of said person's vehicle and a localization of the vehicle window.  identifying a location of the window of the vehicle, wherein the location of the window is used to locate the person. 
     
     
       19. The apparatus according to  claim 11 , wherein the means for processing said plurality of input images in order to extract motion information in a contact-free manner further comprises  further comprising means for performing face detection and hand tracking of the person. 
     
     
       20. The apparatus according to  claim 11 , wherein the means for performing  contact- free  interaction further  comprises means for allowing said person to purchase food items or non-food items, such as pharmaceuticals, or where it is designed to provide services, such as banking, using the contact-free interface.  at least one of a food order, a pharmaceutical order, or a banking transaction. 
     
     
       21. An apparatus for implementing a contact- free interaction, the apparatus comprising:      a display system configured to provide visual information to a person in a vehicle;        an image - capturing system configured to capture one or more images of the vehicle and one or more images of the person interacting with the visual information in a contact - free manner; and        a processing and control system configured to      process the one or more images of the person to identify a selection of the person corresponding to the visual information; and        perform a contact - free interaction with the person based on the selection, wherein the contact - free interaction occurs regardless of whether a window of the vehicle is open or closed.       
     
     
       22. The apparatus of  claim 21 , wherein the visual information is provided on a display, and further wherein a position of the person relative to the visual information is indicated as a visual stimulus on the display. 
     
     
       23. The apparatus of  claim 21 , wherein the processing and control system processes the one or more images of the person using real- time gesture recognition technology.   
     
     
       24. The apparatus of  claim 21 , wherein the processing and control system is further configured to identify a position of the window of the vehicle based at least in part on the one or more images of the vehicle, and further wherein the display system is configured to position the visual information based at least in part on the position of the window. 
     
     
       25. The apparatus of  claim 21 , wherein the processing and control system is further configured to:
   identify a silhouette of the vehicle using background subtraction;        identify a main color of the vehicle based at least in part on the silhouette;        identify a portion of the vehicle which does not include the main color of the vehicle; and        identify the portion of the vehicle as the window of the vehicle using edge detection and geometrical information regarding the vehicle.     
     
     
       26. The apparatus of  claim 21 , wherein the processing and control system is further configured to identify a level of eyesight of the person, and further wherein the display system is configured to position the visual information based at least in part on the level of eyesight. 
     
     
       27. The apparatus of  claim 21 , wherein the one or more images of the person are captured in an interaction range volume, and further wherein the interaction range volume is identified based at least in part on a level of eyesight of the person, a physical dimension of the person, a physical dimension of the vehicle, or a location of the window of the vehicle. 
     
     
       28. The apparatus of  claim 21 , wherein the processing and control system is further configured to determine if the person is looking at the visual information, and further wherein the display system is configured to modify the visual information based at least in part on whether the person is looking at the visual information. 
     
     
       29. A method for implementing a contact- free interaction, the method comprising:      identifying a vehicle and a person in the vehicle with an image - capturing device;        providing visual information to the person through a display;        identifying a hand gesture of the person, wherein the hand gesture does not involve contact with the display;        identifying, based on the hand gesture, a selection from the person corresponding to the visual information; and        implementing a contact - free transaction based on the selection, wherein the contact - free transaction occurs regardless of whether a window of the vehicle is open or closed.     
     
     
       30. The method of  claim 29 , wherein a position of the visual information on the display is based at least in part on a level of eyesight of the person, a physical dimension of the person, a physical dimension of the vehicle, or a location of the window of the vehicle. 
     
     
       31. The method of  claim 29 , further comprising identifying a position of the window based at least in part on a silhouette of the vehicle.