Systems, methods and apparatuses for stereo vision and tracking

Abstract

A system, method and apparatus for stereo vision and tracking with a plurality of coupled cameras and optional sensors.

What is claimed is: 
     
         1 . A stereo vision procurement apparatus for obtaining stereo visual data, comprising:
 a stereo RGB camera;   a depth sensor; and   an RGB-D fusion module,   a processor; and   a plurality of tracking devices to track movement of a subject,   wherein:
 the processor is configured to process data from the tracking devices to form a plurality of sub-features, 
 said sub-features are combined by said FPGA to form a feature to track movements of the subject, 
 each of said stereo RGB camera and said depth sensor are configured to provide pixel data corresponding to a plurality of pixels, 
 said RGB-D fusion module is configured to combine RGB pixel data from said stereo RGB camera and depth information pixel data from said depth sensor to form stereo visual pixel data (SVPD), and 
 said RGB-D fusion module is implemented in an FPGA field-programmable gate array). 
   
     
     
         2 . The apparatus of  claim 1 , further comprising a de-mosaicing module configured to perform a method comprising:
 averaging the RGB pixel data associated with a plurality of green pixels surrounding red and blue sites for R(B) at B-G(R-G) sites or R(B) at R-G(B-G) sites, and   reducing a number of green pixel values from the RGB pixel data to fit a predetermined pixel array (e.g., a 5×5 window) for R(B) at B(R) sites.   
     
     
         3 . The apparatus of  claim 2 , wherein:
 said stereo RGB camera comprises a first camera and a second camera,   each of said first and second cameras being associated with a clock on said FPGA, and   said FPGA including a double clock sampler for synchronizing said clocks of said first and right cameras.   
     
     
         4 . The apparatus of  claim 3 , further comprising:
 a histogram module comprising a luminance calculator for determining a luminance level of at least said RGB pixel data; and   a classifier for classifying said RGB pixel data according to said luminance level, wherein said luminance level is transmitted to said stereo RGB camera as feedback.   
     
     
         5 . The apparatus of  claim 4 , further comprising a white balance module configured to apply a smoothed GW (gray world) algorithm to said RGB pixel data. 
     
     
         6 . The apparatus of  claim 1 , further comprising:
 a biological sensor configured to provide biological data,   wherein:
 said biological sensor is selected from the group consisting of: an EEG sensor, a heartrate sensor, an oxygen saturation sensor, an EKG sensor, or EMG sensor, and a combination thereof, 
 the processor is configured to process the biological data to form a plurality of sub-features, and 
 said sub-features are combined by the FPGA to form a feature. 
   
     
     
         7 . The apparatus of  claim 1 , wherein said FPGA is implemented as a field-programmable gate array (FPGA) comprising a system on a chip (SoC), including an operating system as a SOM (system on module). 
     
     
         8 . The apparatus of  claim 7 , further comprising a CPU SOM for performing overflow operations from said FPGA. 
     
     
         9 . The apparatus of  claim 1 , wherein said tracking devices comprise a plurality of wearable sensors. 
     
     
         10 . The apparatus of  claim 9 , further comprising:
 a multi-modal interaction device in communication with a subject, said multi-modal interaction device comprising said plurality of tracking devices and at least one haptic feedback device,   wherein:
 the processor is configured to process data from the tracking devices to form a plurality of tracking sub-features, and 
 said sub-features are combined by said FPGA to form a feature to track movements of the subject and to provide feedback through said at least one haptic feedback device. 
   
     
     
         11 . The apparatus of  claim 1  further comprising:
 a memory; and 
 wherein said processor is configured to perform a defined set of operations in response to receiving a corresponding instruction selected from an instruction set of codes, and said instruction set of codes include:
 a first set of codes for operating said RGB-D fusion module to synchronize RGB pixel data and depth pixel data, and for creating a disparity map; and 
 a second set of codes for creating a point cloud from said disparity map and said depth pixel data. 
 
 
     
     
         12 . The apparatus of  claim 11 , wherein said point cloud comprises a colorized point cloud. 
     
     
         13 . The apparatus of  claim 1  further comprising:
 a memory; and 
 wherein said processor is configured to perform a defined set of operations for performing any of the functionality recited in  claim 1  in response to receiving a corresponding instruction selected from an instruction set of codes, wherein said codes are stored in said memory. 
 
     
     
         14 . The apparatus of  claim 13 , wherein said processor is configured to operate according to a set of codes selected from the instruction set for a de-noising process for a CFA (color filter array) image according to a W-means process. 
     
     
         15 . The apparatus of  claim 14 , wherein said computational device comprises a second set of codes selected from the instruction set for operating a bad pixel removal process. 
     
     
         16 . A system comprising the apparatus of  claim 1 , further comprising a display for displaying stereo visual data. 
     
     
         17 . A method for processing image information comprising:
 receiving SVPD from the stereo camera of the apparatus of  claim 1 ;   performing RGB preprocessing on the input pixel data to produce preprocessed RGB image pixel data;   using the RGB preprocessed image pixel data in the operation of the stereo camera with respect to at least one of an autogain and an autoexposure algorithm;   rectifying the SVPD so as to control artifacts caused by the lens of the camera; and   calibrating the SVPD so as to prevent distortion of the stereo pixel input data by the lens of the stereo camera.   
     
     
         18 . The method of  claim 17 , further comprising colorizing the preprocessed RGB image pixel data, and creating a disparity map based on the colorized, preprocessed RGB image pixel data. 
     
     
         19 . The method of  claim 18 , wherein calibration comprises matching the RGB pixel image data with depth pixel data. 
     
     
         20 . The method of  claim 19 , wherein the disparity map is created by:
 obtaining depth pixel data from at least one of the stereo pixel input data, the preprocessed RGB image pixel data, and depth pixel data from a depth sensor, and checking differences between stereo images.   
     
     
         21 . The method of  claim 20 , wherein said disparity map, plus depth pixel data from the depth sensor in the form of a calibrated depth map, is combined for the point cloud computation.