Systems, apparatuses, and methods for gesture recognition and interaction

Abstract

Generally discussed herein are systems and apparatuses for gesture-based augmented reality. Also discussed herein are methods of using the systems and apparatuses. According to an example a method may include detecting, in image data, an object and a gesture, in response to detecting the object in the image data, providing data indicative of the detected object, in response to detecting the gesture in the image data, providing data indicative of the detected gesture, and modifying the image data using the data indicative of the detected object and the data indicative of the detected gesture.

What is claimed is: 
     
         1 . A device comprising:
 a camera to capture image data including video;   a display to provide a view of the image data;   processing circuitry coupled to the camera and the display, the processing circuitry configured to:
 identify one or more fingers in the video; 
 recognize a first gesture performed by the identified one or more fingers; and 
 in response to recognizing the first recognized gesture, cause a still image including a first object proximate the first gesture to be provided on the display; 
   the camera is further configured to continue to capture second video while the display is providing the view of the still image; and   the processing circuitry is further configured to:
 extract data corresponding to the identified one or more fingers from the second video; 
 modify the still image to include the extracted one or more fingers from the second video overlaid on the still image; and 
 cause the display to provide a view of the modified still image including the extracted one or more fingers from the second video overlaid thereon. 
   
     
     
         2 . The device of  claim 1 , wherein the processing circuitry is further configured to recognize a second gesture performed by the identified one or more fingers proximate the modified still image and augment the modified still image based on the second gesture. 
     
     
         3 . The device of  claim 1 , wherein the processing circuitry is further configured to modify the still image, in response to the second gesture, with a list of one or more user-selectable operations, which when selected, cause the processing circuitry to modify the image data using a selected operation. 
     
     
         4 . The device of  claim 1 , wherein the processing circuitry is further configured to identify a first object in the image data. 
     
     
         5 . The device of  claim 4 , wherein the processing circuitry is further configured to, in response to recognition of the second gesture, modify the image data by performing a first operation, and modify the image data by performing a second operation, different from the first operation, in response to determining a recognized object is the second object, different from the first object. 
     
     
         6 . The device of  claim 1 , further comprising:
 a microphone to capture sounds;   wherein the processing circuitry is further configured to:
 translate one or more sounds captured by the microphone and provide data indicative of the translated one or more sounds; 
 determine one or more social circumstances of the user based on a location, a speed of the user, a translated sound, one or more objects in the image data, and one or more people in the image data, wherein a social circumstance is one of a plurality of social circumstances including the user of the device exercising, conversing, driving, shopping, eating, and working; 
 modify the image by performing a first operation on the image data based on a first social circumstance of the determined one or more social circumstances, and the second gesture. 
   
     
     
         7 . The device of  claim 6 , wherein the processing circuitry is further configured to receive the data indicative of the first recognized gesture to determine whether the first recognized gesture satisfies a policy including one or more gestures that must be performed before a user is allowed access to the functionality of the device, and in response to determining the policy has been satisfied, provide data indicating a valid authentication procedure has been performed. 
     
     
         8 . A method performed by a device, the method comprising:
 capturing image data including video;   providing a view of the image data;   identifying one or more fingers in the video;   recognizing a first gesture performed by the identified one or more fingers; and   in response to recognizing the first recognized gesture, causing a still image including a first object proximate the first gesture to be provided on the display;   continuing to capture second video while the display is providing the view of the still image;   extracting data corresponding to the identified one or more fingers from the second video;   modifying the still image to include the extracted one or more fingers from the second video overlaid on the still image; and   causing the display to provide a view of the modified still image including the extracted one or more fingers from the second video overlaid thereon.   
     
     
         9 . The method of  claim 8 , further comprising recognizing a second gesture performed by the identified one or more fingers proximate the modified still image and augment the modified still image based on the second gesture. 
     
     
         10 . The method of  claim 8 , further comprising modifying the still image, in response to the second gesture, with a list of one or more user-selectable operations, which when selected, causes modification of the image data using a selected operation. 
     
     
         11 . The method of  claim 8 , further comprising identifying a first object in the image data. 
     
     
         12 . The method of  claim 11 , further comprising, in response to recognition of the second gesture, modifying the image data by performing a first operation, and modifying the image data by performing a second operation, different from the first operation, in response to determining a recognized object is the second object, different from the first object. 
     
     
         13 . The method of  claim 8 , further comprising:
 capturing, by a microphone, sounds;   translating one or more sounds captured by the microphone and provide data indicative of the translated one or more sounds;   determining one or more social circumstances of the user based on a location, a speed of the user, a translated sound, one or more objects in the image data, and one or more people in the image data, wherein a social circumstance is one of a plurality of social circumstances including the user of the device exercising, conversing, driving, shopping, eating, and working; and   modifying the image by performing a first operation on the image data based on a first social circumstance of the one or more social circumstances and the second gesture.   
     
     
         14 . The device of  claim 13 , wherein the processing circuitry is further configured to receive the data indicative of the first recognized gesture to determine whether the first recognized gesture satisfies a policy including one or more gestures that must be performed before a user is allowed access to the functionality of the device, and in response to determining the policy has been satisfied, provide data indicating a valid authentication procedure has been performed. 
     
     
         15 . A non-transitory machine readable medium including instructions that, when executed by a machine, case the machine to perform operations comprising:
 capturing image data including video;   providing a view of the image data;   identifying one or more fingers in the video;   recognizing a first gesture performed by the identified one or more fingers; and   in response to recognizing the first recognized gesture, causing a still image including a first object proximate the first gesture to be provided on the display;   continuing to capture second video while the display is providing the view of the still image;   extracting data corresponding to the identified one or more fingers from the second video;   modifying the still image to include the extracted one or more fingers from the second video overlaid on the still image; and   causing the display to provide a view of the modified still image including the extracted one or more fingers from the second video overlaid thereon.   
     
     
         16 . The non-transitory machine-readable medium of  claim 15 , wherein the operations further comprise recognizing a second gesture performed by the identified one or more fingers proximate the modified still image and augment the modified still image based on the second gesture. 
     
     
         17 . The non-transitory machine-readable medium of  claim 15 , wherein the operations further comprise modifying the still image, in response to the second gesture, with a list of one or more user-selectable operations, which when selected, causes modification of the image data using a selected operation. 
     
     
         18 . The non-transitory machine-readable medium of  claim 15 , wherein the operations further comprise identifying a first object in the image data. 
     
     
         19 . The non-transitory machine-readable medium of  claim 18 , wherein the operations further comprise, in response to recognition of the second gesture, modifying the image data by performing a first operation, and modifying the image data by performing a second operation, different from the first operation, in response to determining a recognized object is the second object, different from the first object. 
     
     
         20 . The non-transitory machine-readable medium of  claim 15 , wherein the operations further comprise:
 capturing, by a microphone, sounds;   translating one or more sounds captured by the microphone and provide data indicative of the translated one or more sounds;   determining one or more social circumstances of the user based on a location, a speed of the user, a translated sound, one or more objects in the image data, and one or more people in the image data, wherein a social circumstance is one of a plurality of social circumstances including the user of the device exercising, conversing, driving, shopping, eating, and working; and   modifying the image by performing a first operation on the image data based on a first social circumstance of the one or more social circumstances and the second gesture.