In-cabin hazard prevention and safety control system for autonomous machine applications

Abstract

In various examples, systems and methods are disclosed that accurately identify driver and passenger in-cabin activities that may indicate a biomechanical distraction that prevents a driver from being fully engaged in driving a vehicle. In particular, image data representative of an image of an occupant of a vehicle may be applied to one or more deep neural networks (DNNs). Using the DNNs, data indicative of key point locations corresponding to the occupant may be computed, a shape and/or a volume corresponding to the occupant may be reconstructed, a position and size of the occupant may be estimated, hand gesture activities may be classified, and/or body postures or poses may be classified. These determinations may be used to determine operations or settings for the vehicle to increase not only the safety of the occupants, but also of surrounding motorists, bicyclists, and pedestrians.

What is claimed is: 
     
       1. A method comprising:
 determining, using one or more neural networks and based at least on image data representative of an image depicting an occupant, a first location associated with a first hand of the occupant and a second location associated with a second hand of the occupant; 
 determining, based at least on one or more of the first location or the second location, an activity associated with the occupant; and 
 performing, based at least on the activity, one or more operations of an autonomous or semi-autonomous machine. 
 
     
     
       2. The method of  claim 1 , further comprising:
 determining, using the one or more neural networks and based at least on the image data, a first angle associated with the first hand and a second angle associated with the second hand, 
 wherein the determining the activity associated with the occupant is further based at least on one or more of the first angle or the second angle. 
 
     
     
       3. The method of  claim 1 , wherein:
 the determining the activity associated with the occupant comprises:
 determining, based at least on the first location, a first activity associated with the first hand; and 
 determining, based at least on the second location, a second activity associated with the second hand; and 
 the performing the one or more operations of the autonomous or semi-autonomous machine is based at least on one or more of the first activity or the second activity. 
 
 
     
     
       4. The method of  claim 3 , further comprising:
 determining a first priority associated with the first activity and a second priority associated with the second activity, 
 wherein the performing the one or more operations of the autonomous or semi-autonomous machine is further based at least on one or more of the first priority or the second priority. 
 
     
     
       5. The method of  claim 1 , further comprising:
 determining, using the one or more neural networks and based at least on the image data, one or more first key points associated with the first hand and one or more second key points associated with the second hand, 
 wherein the determining the first location and the second location is based at least on the one or more first key points and the one or more second key points. 
 
     
     
       6. The method of  claim 1 , wherein the determining the first location and the second location comprises determining, using the one or more neural networks and based at least on the image data, a first bounding shape associated with the first hand and a second bounding shape associated with the second hand, the first bounding shape corresponding to the first location associated with the first hand and the second bounding shape corresponding to the second location associated with the second hand. 
     
     
       7. The method of  claim 1 , wherein:
 the first location associated with the first hand comprises at least one of a first two-dimensional (2D) location associated with the first hand or a first three-dimensional (3D) location associated with the first hand; and 
 the second location associated with the second hand comprises at least one of a second 2D location associated with the second hand or a second 3D location associated with the second hand. 
 
     
     
       8. The method of  claim 1 , wherein the determining the activity associated with the occupant uses one or more second neural networks that are different than the one or more neural networks used to determine the first location and the second location. 
     
     
       9. A system comprising:
 one or more processing units to:
 determine, based at least on image data representative of a first image and a second image, one or more first key points associated with an occupant as depicted by the first image and one or more second key points associated with the occupant as depicted by the second image; 
 determine, using one or more neural networks and based at least on the one or more first key points and the one or more second key points, an activity associated with the occupant; and 
 perform, based at least on the activity, one or more operations of an autonomous or semi-autonomous machine. 
 
 
     
     
       10. The system of  claim 9 , wherein the one or more processing units are further to:
 generate a first pose model based at least on the one or more first key points; and 
 generate a second pose model based at least on the one or more second key points, 
 wherein the determination of the activity is based at least on the first pose model and the second pose model. 
 
     
     
       11. The system of  claim 9 , wherein the one or more processing units are further to:
 determine, based at least on the one or more first key points, a first volume associated with the occupant; and 
 determine, a based at least on the one or more second key points, a second volume associated with the occupant, 
 wherein the determination of the activity is based at least on the first volume and the second volume. 
 
     
     
       12. The system of  claim 9 , wherein the one or more processing units are further to:
 determine, based at least on the one or more first key points, a first representation associated with the occupant; and 
 determine, based at least on the one or more second key points, a second representation associated with the occupant, 
 wherein the determination of the activity is based at least on the first representation and the second representation. 
 
     
     
       13. The system of  claim 9 , wherein the determination of the one or more first key points and the one or more second key points uses one or more second neural networks that are different than the one or more neural networks. 
     
     
       14. The system of  claim 9 , wherein:
 the one or more first key point comprise at least one of one or more first two-dimensional (2D) locations associated with the occupant or one or more first three-dimensional (3D) locations associated with the occupant; and 
 the one or more second key point comprise at least one of one or more second 2D locations associated with the occupant or one or more second 3D locations associated with the occupant. 
 
     
     
       15. The system of  claim 9 , wherein the image data represents a sequence of images captured over a time window, the sequence of images including at least the first image and the second image. 
     
     
       16. The system of  claim 9 , wherein the determination of the activity associated with the occupant comprises:
 determining, using the one or more neural networks and based at least on the one or more first key points and the one or more second key points, a plurality of confidences for a plurality of activities associated with the occupant; and 
 determining, based at least on the plurality of confidences, the activity from the plurality of activities. 
 
     
     
       17. The system of  claim 9 , wherein the one or more processing units are further to:
 determine, based at least on at least a portion of the image data, a location associated with the occupant within the autonomous or semi-autonomous machine, 
 wherein the determination of the activity is further based at least on the location associated with the occupant. 
 
     
     
       18. The system of  claim 9 , wherein the system is comprised in at least one of:
 a control system for the autonomous or semi-autonomous machine; 
 a perception system for the autonomous or semi-autonomous machine; 
 a system for performing simulation operations; 
 a system for performing digital twin operations; 
 a system for performing real-time streaming; 
 a system for generating or presenting virtual reality (VR) content; 
 a system for generating or presenting augmented reality (AR) content; 
 a system for generating or presenting mixed reality (MR) content; 
 a system for performing light transport simulation; 
 a system for performing collaborative content creation for 3D assets; 
 a system for performing deep learning operations; 
 a system implemented using an edge device; 
 a system implemented using a robot; 
 a system for performing conversational AI operations; 
 a system for generating synthetic data; 
 a system incorporating one or more virtual machines (VMs); 
 a system implemented at least partially in a data center; or 
 a system implemented at least partially using cloud computing resources. 
 
     
     
       19. A processor comprising:
 one or more processing units to perform one or more operations of an autonomous or semi-autonomous machine based at least on an activity associated with an occupant of the machine, wherein the activity associated with the occupant is determined using one or more neural networks and based at least on a first bounding shape associated with a first hand represented by image data and a second bounding shape associated with a second hand represented by the image data. 
 
     
     
       20. The processor of  claim 19 , wherein the processor is comprised in at least one of:
 a control system for the autonomous or semi-autonomous machine; 
 a perception system for the autonomous or semi-autonomous machine; 
 a system for performing simulation operations; 
 a system for performing digital twin operations; 
 a system for performing real-time streaming; 
 a system for generating or presenting virtual reality (VR) content; 
 a system for generating or presenting augmented reality (AR) content; 
 a system for generating or presenting mixed reality (MR) content; 
 a system for performing light transport simulation; 
 a system for performing collaborative content creation for 3D assets; 
 a system for performing deep learning operations; 
 a system implemented using an edge device; 
 a system implemented using a robot; 
 a system for performing conversational AI operations; 
 a system for generating synthetic data; 
 a system incorporating one or more virtual machines (VMs); 
 a system implemented at least partially in a data center; or 
 a system implemented at least partially using cloud computing resources.