Apparatus and system for interfacing with computers and other electronic devices through gestures by using depth sensing and methods of use

Abstract

Disclosed herein are systems and methods for gesture capturing, detection, recognition, and mapping them into commands that allow one or more users to interact with electronic games or any electronic device interfaces. Gesture recognition methods, apparatus and system are disclosed from which application developers can incorporate gesture-to-character inputs into their gaming, learning or the like applications. Also herein are systems and methods for receiving 3D data reflecting hand, fingers or other body parts movements of a user, and determining from that data whether the user has performed gesture commands for controlling electronic devices, or computer applications such as games or others.

I claim: 
     
         1 . An apparatus for 3D mapping images of one or more objects, comprising:
 a. an illumination device having a light source that modulates frequency, duty cycle, and intensity, for illuminating the one or more objects;   b. an optical device for projecting one or more object images onto an image sensing device;   c. the image sensing device;   d. a reconfigurable architecture hardware comprising an image capturing device for capturing images of one or more objects from the image sensing device, an information storage device, the reconfigurable hardware for controlling the illumination device and implementing a 3D mapping formula generating a 3D map of one or more objects; and   e. a processor comprising an information storage device, for processing the 3D map of one or more objects.   
     
     
         2 . The apparatus of  claim 1 , wherein the illumination device comprises a controlled-modulated light source which spatially samples the scene in the FOV by gradually illuminating with an optical radiation one or more objects placed in the FOV of the optical device, and which is controlled by the reconfigurable architecture hardware. 
     
     
         3 . The apparatus of  claim 1 , the reconfigurable architecture hardware further comprising a communication device, the hardware implementing the depth image calculation to calculate a depth field of one or more objects, the hardware implementing a finite state machine which controls the image sensing device and the illumination device, and the hardware implementing a second finite state machine for communication with the processor and the external devices. 
     
     
         4 . A system for using a 3D mapping image of one or more objects for controlling electronic devices, comprising:
 a. an illumination device having a light source that modulates frequency, duty cycle, and intensity for illuminating one or more objects;   b. an optical device for projecting one or more object images onto an image sensing device;   c. the image sensing device which produces images based on controlled-modulated light illumination of one or more objects situated in a field of view of the optical device; a reconfigurable architecture hardware controlling the illumination device comprising an image capturing device for capturing images of one or more object from the image sensing devices, implementing a 3D mapping formula for generating a 3D map of the one or more objects; and   d. a processor for processing the 3D map of one or more objects for analyzing gestures in order to interpret them through special image processing methods for mapping them into desired commands to control electronic devices.   
     
     
         5 . The system of  claim 4 , further comprising a movement controller, the hardware of which produces signals which measure the movement of hands or other body parts in motion, based on acceleration and rotation or the body parts movement, and which can be read by a processor for building data to implement a “six degrees of freedom” control of external electronic devices, comprising:
 a. an apparatus of  claim 1 ; 
 b. a 3D mapping of one or more objects; 
 c. a controller which comprises acceleration and rotation transducers; 
 d. a method for fusing the 3D map with the data produced by the controller which comprises acceleration and rotation transducers the combination of the two data sets producing a combined data set permitting thus the control of the environment in 6 degrees of freedom. 
 
     
     
         6 . The system of  claim 4 , further comprising hardware for capturing a voice, and software executed in the processor for adding voice understanding and interpretation for further controlling electronic devices. 
     
     
         7 . The system of  claim 4 , wherein the controlled electronic device is a computer, a game console, a TV set, or a set-top box, or any mobile device such that the system sends controls to the electronic device. 
     
     
         8 . The system of  claim 4 , further comprising a plurality of systems communicating over a network allowing two or more system users to interact in controlling electronic devices such that the users can collaborate on controlling in concert or competitively the electronic device or the application which executes on that electronic device. 
     
     
         9 . The system of  claim 4 , further comprising a communication device for sending the result of the processing of the 3D map to the controlled device, the controlled device being any device which can be controlled locally or remotely through a communication protocol specific to the electronic device 
     
     
         10 . A method of illuminating one or more objects, comprising the steps of:
 a. controlling an illumination device to project a controlled train of pulses that are modulated for frequency, duty cycle, and intensity onto the one or more objects;   b. adapting the pulses to the distance of one or more objects from the center of the optical device; and   c. controlling image sensing device parameters according to parameters of the modulated pulses.   
     
     
         11 . The method of  claim 10 , further comprising the steps of:
 a. establishing illumination device parameters by providing a 3D map to a processor and receiving from the processor the illuminating device parameters needed;   b. using data obtained from the processor for calculating parameters of the depth map calculation algorithm; and   c. establishing communication with the processor and with an electronic device.   
     
     
         12 . A method, comprising the steps of:
 a. detecting and recognizing representative features of a face of one or more objects where the object placed in front of the imaging device is a human, which is executed by the reconfigurable architecture hardware; and   b. identifying the face of one or more humans present in the field of view and using the identified part of the human to further process other parts of interest from the body of the human object.   
     
     
         13 . The method of  claim 12 , further comprising the processing of the depth images for identifying body or component parts of one or more objects comprising the steps of:
 a. obtaining depth image data from a reconfigurable architecture hardware and;   b. processing the data to obtain characteristic features;   c. applying corresponding calculation methods identifying body or component parts of one or more objects for real-time matching and tracking of gestures and body parts posture or movements; and   d. mapping a meaning of the gesture or body parts posture or movements onto an action to be transmitted to an external electronic device.   
     
     
         14 . The method of  claim 12 , further comprising the steps of:
 a. identifying gestures made by using hands and fingers;   b. identifying and separating right and left hand wrists;   c. identifying one or more fingers by applying adequate image processing methods, such as building convex, or concave polygons, or applying classification methods or alike;   d. labeling each finger;   e. tracking the fingers;   f. identifying and interpreting a hand and finger gesture; and   g. mapping the meaning of the hand and finger gesture onto an interaction with an electronic device.   
     
     
         15 . The method of  claim 12 , further comprising the steps of:
 a. identifying gestures made by either using hands, fingers or body or body parts posture or movements to create a meaning for any gesture made by the user;   b. using the face identification, the body or body parts identification or the hand and finger identification results to detect, identify, and interpret a gesture which is a combination of face, body or body parts, and hand and fingers movements; and   c. mapping the result of identifying the meaning of the gesture or body parts movements onto an action to be transmitted to the external electronic device.   
     
     
         16 . The method of  claim 15 , further comprising the steps of:
 a. capturing and processing the 3D map image for detecting, identifying, and interpreting the gesture or body parts movements of one or many objects which are present in the field of view of an optical device;   b. capturing data produced by a controller containing other movement detection sensors;   c. fusing the 3D mapping device data with the data produced by the controller; and   d. mapping the fused data in control signals for controlling electronic devices.   
     
     
         17 . The method of  claim 15 , further comprising the steps of:
 a. recognizing the gesture made by one or more objects as part of a gesture language stored in an information storage device of the processor;   b. having defined a gesture language made out by a sequence of gestures, each of which represents the meaning of a natural language word or a phrase, or a sequence of gesture statements, sequence which can be either predefined or assembled by a user of the system;   c. parsing gestures stored in the information storage of the processor which interprets the gestures defined by the gesture language and assembles them into a command to be sent to electronic devices connected to the system; and   d. assembling the gesture language out of the predefined gestures representing cues or from user-defined gestures by means of a graphical user interface.   
     
     
         18 . The method of  claim 18 , further comprising the steps of;
 a. allowing the user to define its own gesture or body parts movements comprising the steps of:   b. detecting, identifying, interpreting, and extracting the cue of the new gesture as defined by the user;   c. integrating the new gesture in the language parser; and   d. storing the new gesture in information storage device.