Motion-Assisted Visual Language for Human Computer Interfaces

Abstract

Embodiments of the invention recognize human visual gestures, as captured by image and video sensors, to develop a visual language for a variety of human computer interfaces. One embodiment of the invention provides a computer-implement method for recognizing a visual gesture portrayed by a part of human body such as a human hand, face or body. The method includes steps of receiving the visual signature captured in a video having multiple video frames, determining a gesture recognition type from multiple gesture recognition types including shaped-based gesture, position-based gesture, motion-assisted and mixed gesture that combining two different gesture types. The method further includes steps of selecting a visual gesture recognition process based on the determined gesture type and applying the selected visual gesture recognition process to the multiple video frames capturing the visual gesture to recognize the visual gesture.

What is claimed is: 
     
         1 . A computer-implemented method for recognizing a visual gesture comprising:
 receiving a visual gesture formed by a part of a human body, the visual gesture captured by a video having a plurality of video frames;   determining a region of interest (ROI) in the plurality of video frames of the video based in part on motion vectors associated with motion of the part of the human body;   applying a visual gesture recognition process to the plurality of video frames;   determining a plurality of features of an object within the ROI based on the applied visual gesture recognition process, the object comprising at least a part of the visual gesture, the plurality of features comprising at least two of a centroid, a shape, and a size; and   determining variations in at least one of the plurality of features, the at least one feature changing according to motion of the object in the plurality of video frames in a motion model, with a learning based tracking process comprising a plurality of functions performed simultaneously, wherein the plurality of functions comprises at least two of:
 an object tracking function using motion estimation in the motion model and employing an estimation error metric comprising one of a sum of absolute differences (SAD) and a normalized correlation coefficient (NCC); 
 an object feature learning function that automatically learns features of objects within the ROI, the features including at least one of size, shape, centroids, statistics, and edges; and 
 an object detection function comprising at least one technique selected from a group consisting of:
 feature extraction employing one of edge analysis, spatial transforms, background subtraction, and neural networks; 
 feature analysis employing one of clustering, vector quantization, and neural networks; and 
 feature matching employing signal matching using one of similarity metrics, neural networks, support vector machines, and maximum posteriori probability; and 
 
   deriving multi-dimensional information of the visual gesture based on the analysis of the determined variations.