Vehicle Occupant Monitoring Using Infrared Imaging

Abstract

Methods and systems for monitoring vehicle occupants using infrared and other sensors are disclosed. The systems may use infrared sensor data to identify each vehicle occupant using biometric signatures (such as heartbeat or facial recognition). Vehicle occupant characteristics may be determined that include skeletal characteristics of the occupants. Further monitoring of the occupants using sensor data may occur until an abnormal situation is detected. Abnormal situations may include medical emergencies, driver impairment, security threats, or similar situations requiring correcting action. The system may then determine and implement an appropriate response to the abnormal situation. Such responses may include generating alerts, adjusting vehicle environmental controls, taking control of operation of the vehicle (such as an autonomous or semi-autonomous vehicle), or initiating wireless communication with an outside party, such as an emergency service.

1 . A computer-implemented method for monitoring a vehicle occupant of a vehicle, the method comprising, via one or more processors;
 receiving, from one or more imaging sensors disposed within the vehicle, imaging sensor data regarding the vehicle occupant;   determining one or more vehicle occupant characteristics for the vehicle occupant based upon the received imaging sensor data, wherein the one or more vehicle occupant characteristics include (i) one or more skeletal characteristics of the vehicle occupant, including at least one joint and a plurality of skeletal segments of at least one arm of the vehicle occupant, and (ii) one or more muscular characteristics of a muscle group associated with at least one of the skeletal segments of the vehicle occupant based at least in part upon the plurality of skeletal segments, the one or more muscular characteristics including at least one of temperature or flushness of the muscle group;   determining whether an abnormal situation exists based upon the one or more determined vehicle occupant characteristics associated with the vehicle occupant, including the one or more skeletal characteristics and the one or more muscular characteristics, wherein the abnormal situation relates to one or more of the following types of abnormal situations: a medical emergency, a health risk, an accident risk, an impairment of the vehicle occupant, or a security threat; and   causing one or more responses to the abnormal situation to be implemented.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein the one or more skeletal characteristics indicate the position of a plurality of segments of the vehicle occupant's body, wherein the plurality of segments include the vehicle occupant's head and at least a portion of a limb of the vehicle occupant. 
     
     
         3 . The computer-implemented method of  claim 1 , further comprising:
 identifying the vehicle occupant by comparing the one or more determined vehicle occupant characteristics with data regarding characteristics stored in a user profile.   
     
     
         4 . The computer-implemented method of  claim 3 , wherein determining whether the abnormal situation exists includes determining whether the one or more determined vehicle occupant characteristics are beyond a baseline range for the vehicle occupant based upon the data regarding the characteristics stored in the user profile of the vehicle occupant. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein the one or more responses include controlling vehicle operation by an on-board computer system. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein the one or more responses include adjusting an environmental condition within the vehicle. 
     
     
         7 . The computer-implemented method of  claim 1 , wherein the one or more responses include communicating a message to an emergency response service. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein the one or more responses include terminating vehicle operation. 
     
     
         9 . The computer-implemented method of  claim 1 , wherein the one or more responses include communicating sensor data to one or more computing devices associated with emergency response personnel. 
     
     
         10 . The computer-implemented method of  claim 1 , wherein the one or more imaging sensors include one or more infrared sensors disposed within the vehicle. 
     
     
         11 . A computer system for monitoring a vehicle occupant of a vehicle, the computer system comprising:
 one or more processors;   one or more imaging sensors disposed within the vehicle and communicatively connected to the one or more processors; and   a program memory coupled to the one or more processors and storing executable instructions that when executed by the one or more processors cause the computer system to:
 receive imaging sensor data regarding the vehicle occupant from the one or more imaging sensors; 
 determine one or more vehicle occupant characteristics for the vehicle occupant based upon the received imaging sensor data, wherein the one or more vehicle occupant characteristics include (i) one or more skeletal characteristics of the vehicle occupant, including at least one joint and a plurality of skeletal segments of at least one arm of the vehicle occupant, and (ii) one or more muscular characteristics of a muscle group associated with at least one of the skeletal segments of the vehicle occupant based at least in part upon the plurality of skeletal segments, the one or more muscular characteristics including at least one of temperature or flushness of the muscle group; 
 determine whether an abnormal situation exists based upon the one or more determined vehicle occupant characteristics associated with the vehicle occupant, including the one or more skeletal characteristics and the one or more muscular characteristics, wherein the abnormal situation relates to one or more of the following types of abnormal situations: a medical emergency, a health risk, an accident risk, an impairment of the vehicle occupant, or a security threat; and 
 cause one or more responses to the abnormal situation to be implemented. 
   
     
     
         12 . The computer system of  claim 11 , wherein the one or more skeletal characteristics indicates the position of a plurality of segments of the vehicle occupant's body, wherein the plurality of segments include the vehicle occupant's head and at least a portion of a limb of the vehicle occupant. 
     
     
         13 . The computer system of  claim 11 , wherein the program memory further stores executable instructions that cause the computer system to:
 identify the vehicle occupant based upon the one or more determined vehicle occupant characteristics by comparing the one or more determined vehicle occupant characteristics with data regarding characteristics stored in a user profile.   
     
     
         14 . The computer system of  claim 13 , wherein the executable instructions that cause the computer system to determine whether the abnormal situation exists further cause the computer system to determine whether the one or more determined vehicle occupant characteristics are beyond a baseline range for the vehicle occupant based upon the data regarding the characteristics stored in the user profile of the vehicle occupant. 
     
     
         15 . The computer system of  claim 11 , wherein the one or more responses include controlling vehicle operation by an on-board computer system. 
     
     
         16 . The computer system of  claim 11 , wherein the one or more responses include adjusting an environmental condition within the vehicle. 
     
     
         17 . The computer system of  claim 11 , wherein the one or more responses include communicating a message to an emergency response service. 
     
     
         18 . The computer system of  claim 11 , wherein the one or more responses include terminating vehicle operation. 
     
     
         19 . The computer system of  claim 11 , wherein the one or more imaging sensors include one or more infrared sensors disposed within the vehicle. 
     
     
         20 . A tangible, non-transitory computer-readable medium storing instructions for monitoring a vehicle occupant of a vehicle that, when executed by at least one processor of a computer system, cause the computer system to:
 receive imaging sensor data regarding the vehicle occupant from one or more imaging sensors disposed within the vehicle;   determine one or more vehicle occupant characteristics for the vehicle occupant based upon the received imaging sensor data, wherein the one or more vehicle occupant characteristics include (i) one or more skeletal characteristics of the vehicle occupant, including at least one joint and a plurality of skeletal segments of at least one arm of the vehicle occupant, and (ii) one or more muscular characteristics of a muscle group associated with at least one of the skeletal segments of the vehicle occupant based at least in part upon the plurality of skeletal segments, the one or more muscular characteristics including at least one of temperature or flushness of the muscle group;   determine whether an abnormal situation exists based upon the one or more determined vehicle occupant characteristics associated with the vehicle occupant, including the one or more skeletal characteristics and the one or more muscular characteristics, wherein the abnormal situation relates to one or more of the following types of abnormal situations: a medical emergency, a health risk, an accident risk, an impairment of the vehicle occupant, or a security threat; and   cause one or more response to the abnormal situation to be implemented.