Device for identifying and tracking multiple humans over time

Abstract

A system recognizes human beings in their natural environment, without special sensing devices attached to the subjects, uniquely identifies them and tracks them in three dimensional space. The resulting representation is presented directly to applications as a multi-point skeletal model delivered in real-time. The device efficiently tracks humans and their natural movements by understanding the natural mechanics and capabilities of the human muscular-skeletal system. The device also uniquely recognizes individuals in order to allow multiple people to interact with the system via natural movements of their limbs and body as well as voice commands/responses.

What is claimed: 
     
       1. A system for analyzing a user in a capture area, comprising:
 a processor; and 
 a memory communicatively coupled to the processor when the system is operational, the memory bearing processor-executable instructions that, when executed on the processor, cause the system at least to:
 receive depth data for the capture area; 
 receive color data for the capture area; 
 determine a mesh human model of the user in the capture area based on the depth data, the user having a first body part and a second body part, the mesh human model comprising a representation of the first body part and a representation of the second body part of the user; and 
 determine an orientation of the first body part of the user relative to the second body part of the user based on the color data. 
 
 
     
     
       2. The system of  claim 1 , wherein the mesh human model comprises one or more polygonal meshes. 
     
     
       3. The system of  claim 2 , wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:
 receive second color data for the capture area; 
 determine a second orientation of the first body part of the user relative to the second body part of the user based on the second color data; 
 determine that the second orientation of the first body part of the user relative to the second body part of the user violates a bending limit of at least one of the one or more polygonal meshes; and 
 modify an orientation of the representation of the first body part of the user relative to the representation of the second body part of the user in the mesh human model based on the bending limit. 
 
     
     
       4. The system of  claim 1 , wherein the instructions that, when executed on the processor, cause the system at least to determine the orientation of the first body part of the user relative to the second body part of the user based on the color data, further cause the system at least to:
 determine that an orientation of the user comprises the user facing toward a capture device in response to determining that a face of the user is present in the color data. 
 
     
     
       5. The system of  claim 1 , wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:
 determine a location of the user in the capture area based on the depth data. 
 
     
     
       6. The system of  claim 5 , wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:
 track the user over time based on known mechanics of a human muscular-skeletal system. 
 
     
     
       7. The system of  claim 1 , wherein the memory further bears instructions that, when executed on the processor, cause the system at least to:
 track movement of the user over time based on the mesh human model. 
 
     
     
       8. The system of  claim 7 , wherein when the memory further bears instructions that, when executed on the processor, cause the system at least to:
 determine that the system is unable to track movement of the user from the depth data without the color data; and 
 track movement of the user based on the depth data supplemented with the color data. 
 
     
     
       9. The system of  claim 1 , wherein the instructions that, when executed on the processor, cause the system at least to determine the orientation of the first body part of the user relative to the second body part of the user based on the color data, further cause the system at least to:
 determine that an orientation of the user comprises the user facing away from a capture device in response to determining that a threshold amount of hair color of the user or less than a threshold amount of facial features of the user is present in the color data. 
 
     
     
       10. The system of  claim 1 , wherein the color data comprises black and white or grayscale colors. 
     
     
       11. A method for tracking a human user in a capture area, comprising:
 receiving depth data for the capture area; 
 receiving color data for the capture area; 
 determining a mesh human model of the user in the capture area based on the depth data, the mesh human model comprising a first body part and a second body part of the user; and 
 determining an orientation of the first body part of the user relative to the second body part of the user based on the color data. 
 
     
     
       12. The method of  claim 11 , wherein the mesh human model comprises a wireframe mesh. 
     
     
       13. The method of  claim 12 , further comprising:
 receiving second color data for the capture area; 
 determining a second orientation of the first body part of the user relative to the second body part of the user based on the second color data; 
 determining that the second orientation of the first body part of the user relative to the second body part of the user violates a bending limit of the wireframe mesh; and 
 modifying an orientation of the representation of the first body part of the user relative to the representation of the second body part of the user in the mesh human model based on the bending limit. 
 
     
     
       14. The method of  claim 11 , further comprising:
 receiving sound data from at least one microphone; and 
 recognizing and tracking movement of the user in the capture area over time based on a combination of at least two of the depth data, the color data and the sound data. 
 
     
     
       15. The method of  claim 11 , further comprising:
 tracking movement of the user over time based on known mechanics of a human muscular-skeletal system. 
 
     
     
       16. The method of  claim 15 , further comprising:
 determining, based on the tracking, a pose of the mesh human model by applying one or more forces to a vertex of a triangle of the mesh human model. 
 
     
     
       17. The method of  claim 11 , further comprising:
 determining, by one or more computers, that the one or more computers are unable to track movement of the user from the depth data without the color data; and 
 tracking, by the one or more computers, movement of the user based on the depth data supplemented with the color data. 
 
     
     
       18. A computer-readable storage device bearing computer-executable instructions that, when executed by a computer, causing the computer to perform operations comprising:
 receiving depth data for a user in a capture area; 
 receiving color data for the capture area; 
 determining a mesh human model of the user in the capture area based on the depth data, the mesh human model comprising a first body part and a second body part of the user; and 
 determining an orientation of the first body part of the user relative to the second body part of the user based on the color data. 
 
     
     
       19. The computer-readable storage device of  claim 18 , further bearing computer-executable instructions that, when executed by the computer, cause the computer to perform operations comprising:
 shaping, based at least on the depth data, the mesh human model to represent a body shape of the user. 
 
     
     
       20. The computer-readable storage device of  claim 18 , further bearing computer-executable instructions that, when executed by the computer, cause the computer to perform operations comprising:
 determining, based at least on the orientation of the first body part relative to the second body part, one or more polygons of the mesh human model collide; 
 determining, based at least on the collisions, a constraint of where a representation of the first body part may be located relative to the second body part and the location of the second body part; and 
 modifying, based on the constraint, a location of the representation of the first body part of the user relative to a representation of the second body part of the user in the mesh human model.