3D skeleton detection for in-cabin automotive application

Abstract

A monitoring system for a vehicle includes at least one imaging device configured to capture a first image type and a second image type in a sequence. A first illumination source is configured to emit a flood illumination captured by the at least one imaging device in the first image type. A second illumination source is configured to emit an illumination pattern captured by the at least one imaging device in the second image type. At least one processor is configured to extract a 2-dimensional (“2D”) skeletal representation of a vehicle occupant from the first image type, measure a depth of the 2D skeletal representation with the second image type, and extrapolate a 3-dimensional (“3D”) skeletal representation of the vehicle occupant.

What is claimed is: 
     
         1 . A monitoring system for a vehicle comprising:
 at least one imaging device configured to capture a first image type and a second image type in a sequence;   a first illumination source configured to emit a flood illumination captured by the at least one imaging device in the first image type;   a second illumination source configured to emit an illumination pattern captured by the at least one imaging device in the second image type; and   at least one processor configured to:
 extract a 2-dimensional (“2D”) skeletal representation of a vehicle occupant from the first image type; 
 measure a depth of the 2D skeletal representation with the second image type; and 
 extrapolate a 3-dimensional (“3D”) skeletal representation of the vehicle occupant. 
   
     
     
         2 . The monitoring system of  claim 1 , wherein the flood illumination and the illumination pattern are substantially within an infrared spectrum. 
     
     
         3 . The monitoring system of  claim 1 , wherein the flood illumination is substantially within a visible spectrum and the illumination pattern is substantially within an infrared spectrum. 
     
     
         4 . The monitoring system of  claim 1 , wherein the at least one processor is further configured to:
 detect that an airbag is needed because of a crash event;   determine a distance of the 3D skeletal representation from an airbag deployment surface; and   generate an instruction to deploy the airbag from the airbag deployment surface at a predetermined distance between the 3D skeletal representation and the airbag deployment surface.   
     
     
         5 . The monitoring system of  claim 1 , wherein the at least one processor is further configured to:
 detect when a hand of the 3D skeletal representation of the vehicle occupant is in a gesture position;   recognize a hand gesture of the hand; and   control a setting of the vehicle in accordance with the hand gesture.   
     
     
         6 . The monitoring system of  claim 5 , wherein the at least one processor is further configured to:
 recognize the hand gesture as a movement of the hand in a direction; and   distinguish the movement of the hand in the direction from a plurality of directions, each direction of the plurality of directions corresponding to a different control setting.   
     
     
         7 . The monitoring system of  claim 5 , wherein the at least one processor is further configured to:
 recognize the hand gesture as a movement of the hand in a pattern; and   distinguish the pattern from a plurality of patterns, each pattern of the plurality of patterns corresponding to a different control setting.   
     
     
         8 . The monitoring system of  claim 5 , wherein the at least one processor is further configured to:
 recognize a position of at least one finger on the hand as the hand gesture; and   control a setting of the vehicle in accordance with the position of the at least one finger.   
     
     
         9 . The monitoring system of  claim 8 , wherein the position of the at least one finger includes a position of at least two fingers. 
     
     
         10 . The monitoring system of  claim 5 , wherein the gesture position is within a threshold distance from the at least one imaging device. 
     
     
         11 . The monitoring system of  claim 1 , wherein the depth in the second image type is obtained under the principles of at least one of time-of-flight or structured light. 
     
     
         12 . A monitoring system for a vehicle comprising:
 at least one imaging device configured to capture a first image type and a second image type;   the first image type including a 2-dimensional (“2D”) capture of a flood illumination on a vehicle occupant;   the second image type including depth information of the vehicle occupant; and   at least one processor configured to:
 extract a 2-dimensional (“2D”) skeletal representation of a vehicle occupant from the first image type; 
 measure a depth of the 2D skeletal representation with the depth information; 
 extrapolate a 3-dimensional (“3D”) skeletal representation of the vehicle occupant; 
   detect when a hand of the 3D skeletal representation of the vehicle occupant is in a gesture position;   recognize a first hand gesture of the hand; and   generate a menu in accordance with the first hand gesture.   
     
     
         13 . The monitoring system of  claim 12 , wherein the at least one processor is further configured to:
 recognize a second hand gesture of the hand after the menu has been generated; and   scroll through a plurality of settings on the menu in accordance with the second hand gesture.   
     
     
         14 . The monitoring system of  claim 13 , wherein the second hand gesture is different than the first hand gesture and corresponds to movement of the hand in one of a first direction or a first pattern. 
     
     
         15 . The monitoring system of  claim 13 , wherein the at least one processor is further configured to:
 recognize a third hand gesture of the hand after the plurality of settings have been scrolled to a desired setting; and   select the desired setting on the menu in accordance with the third hand gesture.   
     
     
         16 . The monitoring system of  claim 15 , wherein the first hand gesture, the second hand gesture, and the third hand gesture are different. 
     
     
         17 . The monitoring system of  claim 15 , wherein the menu is generated on a full display rearview mirror assembly. 
     
     
         18 . The monitoring system of  claim 15 , wherein the depth information in the second image type is obtained under the principles of at least one of time-of-flight, stereo vision, or structured light. 
     
     
         19 . A monitoring system for a vehicle comprising:
 at least one imaging device configured to capture a first image type and a second image type;   the first image type including a 2-dimensional (“2D”) capture of a flood illumination on a vehicle occupant;   the second image type including depth information of the vehicle occupant; and   at least one processor configured to:
 extract a 2-dimensional (“2D”) skeletal representation of a vehicle occupant from the first image type; 
 measure a depth of the 2D skeletal representation with the depth information; 
 extrapolate a 3-dimensional (“3D”) skeletal representation of the vehicle occupant; 
   detect when a hand of the 3D skeletal representation of the vehicle occupant is in a gesture position within a threshold distance of the at least one imaging device;   recognize a first hand gesture of the hand; and   generate a menu on a full display rearview mirror assembly in accordance with the first hand gesture.   
     
     
         20 . The monitoring system of  claim 19 , wherein the depth information in the second image type is obtained under the principles of at least one of time-of-flight, stereo vision, or structured light.