Human posture determination method and mobile machine using the same

Abstract

Human posture determination is disclosed. Human posture is determined by obtaining range image(s) through a range camera, detecting key points of an estimated skeleton of a human in color data of the range image(s) and calculating positions of the detected key points based on depth data of the range image(s), choosing a feature map from a set of predefined feature maps based on the detected key points among a set of predefined key points, obtaining two features of a body of the human corresponding to the chosen feature map based on the positions of the detected key points, and determining a posture of the human according to the two features in the chosen feature map.

What is claimed is: 
     
       1. A human posture determination method, comprising:
 obtaining, through a range camera, one or more range images, wherein the one or more range images include color data and depth data; 
 detecting key points of an estimated skeleton of a human in the color data and calculating positions of the detected key points based on the depth data, wherein the estimated skeleton has a set of predefined key points; 
 choosing a feature map from a set of predefined feature maps based on the detected key points among the predefined key points; 
 obtaining two features of a body of the human corresponding to the chosen feature map based on the positions of the detected key points; and 
 determining a posture of the human according to the two features in the chosen feature map; 
 wherein when the detected key points comprise at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points, a second feature map with the two features of a body ratio and an upper body angle is chosen from the set of predefined feature maps, and the second feature map comprises: a first threshold curve for distinguishing a standing posture and a sitting posture, and a second threshold curve for distinguishing a lying posture, the standing posture, and the sitting posture; and 
 when the determined posture is not the lying posture and the detected key points comprise all the predefined key points, a first feature map with the two features of an internal angle and the body ratio is chosen from the set of predefined feature maps, and the first feature map comprises a threshold curve for distinguishing the standing posture and the sitting posture. 
 
     
     
       2. The method of  claim 1 ,
 wherein when the first feature map is chosen, the obtaining the two features of the body of the human corresponding to the chosen feature map based on the positions of the detected key points comprises: 
 in response to the detected key points including all the predefined key points, obtaining a detected internal angle of the body of the human and a detected body ratio of the body of the human based on the positions of the detected key points; and 
 the determining the posture of the human according to the two features in the chosen feature map comprises: 
 in response to the detected key points including all the predefined key points, determining the posture of the human based on a position of an intersection of the detected internal angle and the detected body ratio in the first feature map. 
 
     
     
       3. The method of  claim 2 , wherein the obtaining the internal angle and the body ratio based on the positions of the key points comprises:
 obtaining a middle position p h  between the positions of two hip key points among the detected key points; 
 obtaining an upper body plane based on the middle position p h  and the positions of two shoulder key points among the detected key points; 
 obtaining a lower body plane based on the middle position p h  and the positions of two knee key points among the detected key points; 
 obtaining an angle between the upper body plane and the lower body plane to take as the internal angle; 
 obtaining a ratio between a lower body height h low  and an upper body height h up  to take as the body ratio, wherein h up =p s ·y−p h ·y and h low =p h ·y−p k ·y, p s  is a middle position between the positions of the two shoulder key points, p s ·y is a y coordinate of p s , p h ·y is a y coordinate of p h , p k  is a middle position between the positions of the two knee key points, and p k ·y is a y coordinate of p k ; and 
 the determining the posture of the human based on the position of the intersection of the internal angle and the body ratio in the first feature map comprises: 
 determining the posture of the human according to the position of the intersection of the internal angle and the body ratio in the first feature map with respect to the threshold curve for distinguishing the standing posture and the sitting posture. 
 
     
     
       4. The method of  claim 1 ,
 wherein when the second feature map is chosen, the obtaining the two features of the body of the human corresponding to the chosen feature map based on the positions of the detected key points comprises: 
 in response to the detected key points including at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points, obtaining a detected body ratio of the body of the human and a detected upper body angle of the body of the human based on the positions of the detected key points; and 
 the determining the posture of the human according to the two features in the chosen feature map comprises: 
 in response to the detected key points including at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points, determining the posture of the human based on a position of an intersection of the detected body ratio and the detected upper body angle in the second feature map. 
 
     
     
       5. The method of  claim 4 , wherein the obtaining the body ratio and the upper body angle based on the positions of the detected key points comprises:
 obtaining a middle position p h  between the positions of two hip key points among the detected key points; 
 obtaining a ratio between a lower body height h low  and an upper body height h up  to take as the body ratio, wherein h up =p s ·y−p h ·y and h low =p h ·y p k ·y, p s  is a middle position between the positions of two shoulder key points, p s ·y is a y coordinate of p s , p h ·y is a y coordinate of p h , p k  is a middle position between the positions of two knee key points, and p k ·y is a y coordinate of p k ; 
 obtaining an angle between a vector {right arrow over (v)} hs  between the middle position p h  and the middle position p s  and a unit vector {right arrow over (y)} in y-direction to take as the upper body angle; and 
 the determining the posture of the human based on the position of the intersection of the body ratio and the upper body angle in the second feature map comprises: 
 determining the posture of the human according to the position of the intersection of the body ratio and the upper body angle in the second feature map with respect to the first threshold curve for distinguishing the standing posture and the sitting posture and the second threshold curve for distinguishing the lying posture, the standing posture, and the sitting posture. 
 
     
     
       6. The method of  claim 5 , further comprising:
 in response to the detected key points including one shoulder key point, obtaining the position of another of the two shoulder key points based on the position of the detected shoulder key point; 
 in response to the detected key points including one hip key point, obtaining the position of another of the two hip key points based on the position of the detected hip key point; and 
 in response to the detected key points including one knee key point, obtaining the position of another of the two knee key points based on the position of the detected knee key point. 
 
     
     
       7. The method of  claim 1 , further comprising:
 in response to the detected key points including a plurality of head key points among the predefined key points, calculating a head height H head  based on the head key points; and 
 determining the posture of the human by comparing the head height H head  with a first head height threshold for distinguishing a standing posture and a sitting posture and a second head height threshold for distinguishing the sitting posture and a lying posture. 
 
     
     
       8. The method of  claim 1 , wherein the predefined key points include two eye key points, a nose key point, two ear key points, a neck key point, two shoulder key points, two elbow key points, two hand key points, two hip key points, two knee key points, and two foot key points, and wherein the two eye key points, the nose key point, and the two ear key points are referred to as head key points. 
     
     
       9. A mobile machine, comprising:
 a range camera; 
 one or more processors; and 
 a memory storing one or more programs configured to be executed by the one or more processors, wherein the one or more programs include instructions to: 
 obtain, through the range camera, one or more range images, wherein the one or more range images include color data and depth data; 
 detect key points of an estimated skeleton of a human in the color data and calculating positions of the detected key points based on the depth data, wherein the estimated skeleton has a set of predefined key points; 
 choose a feature map from a set of predefined feature maps based on the detected key points among the predefined key points; 
 obtain two features of a body of the human corresponding to the chosen feature map based on the positions of the detected key points; and 
 determine a posture of the human according to the two features in the chosen feature map; 
 wherein when the detected key points comprise at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points, a second feature map with the two features of a body ratio and an upper body angle is chosen from the set of predefined feature maps, and the second feature map comprises: a first threshold curve for distinguishing a standing posture and a sitting posture, and a second threshold curve for distinguishing a lying posture, the standing posture, and the sitting posture; and 
 when the determined posture is not the lying posture and the detected key points comprise all the predefined key points, a first feature map with the two features of an internal angle and the body ratio is chosen from the set of predefined feature maps, and the first feature map comprises a threshold curve for distinguishing the standing posture and the sitting posture. 
 
     
     
       10. The mobile machine of  claim 9 ,
 wherein when the first feature map is chosen, the obtaining the two features of the body of the human corresponding to the chosen feature map based on the positions of the detected key points comprises: 
 in response to the detected key points including all the predefined key points, obtaining a detected internal angle of the body of the human and a detected body ratio of the body of the human based on the positions of the detected key points; and 
 the determining the posture of the human according to the two features in the chosen feature map comprises: 
 in response to the detected key points including all the predefined key points, determining the posture of the human based on a position of an intersection of the detected internal angle and the detected body ratio in the first feature map. 
 
     
     
       11. The mobile machine of  claim 10 , wherein the obtaining the internal angle and the body ratio based on the positions of the key points comprises:
 obtaining a middle position p h  between the positions of two hip key points among the detected key points; 
 obtaining an upper body plane based on the middle position p h  and the positions of two shoulder key points among the detected key points; 
 obtaining a lower body plane based on the middle position p h  and the positions of two knee key points among the detected key points; 
 obtaining an angle between the upper body plane and the lower body plane to take as the internal angle; 
 obtaining a ratio between a lower body height h low  and an upper body height h up  to take as the body ratio, wherein h up =p s ·y−p h ·y and h low =p h ·y−p k ·y, p s  is a middle position between the positions of the two shoulder key points, p s ·y is a y coordinate of p s , p h ·y is a y coordinate of p h , p k  is a middle position between the positions of the two knee key points, and p k ·y is a y coordinate of p k ; and 
 the determining the posture of the human based on the position of the intersection of the internal angle and the body ratio in the first feature map comprises: 
 determining the posture of the human according to the position of the intersection of the internal angle and the body ratio in the first feature map with respect to the threshold curve for distinguishing the standing posture and the sitting posture. 
 
     
     
       12. The mobile machine of  claim 9 ,
 wherein when the second feature map is chosen, the obtaining the two features of the body of the human corresponding to the chosen feature map based on the positions of the detected key points comprises: 
 in response to the detected key points including at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points, obtaining a detected body ratio of the body of the human and a detected upper body angle of the body of the human based on the positions of the detected key points; and 
 the determining the posture of the human according to the two features in the chosen feature map comprises: 
 in response to the detected key points including at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points, determining the posture of the human based on a position of an intersection of the detected body ratio and the detected upper body angle in the second feature map. 
 
     
     
       13. The mobile machine of  claim 12 , wherein the obtaining the body ratio and the upper body angle based on the positions of the detected key points comprises:
 obtaining a middle position p h  between the positions of two hip key points among the detected key points; 
 obtaining a ratio between a lower body height h low  and an upper body height h up  to take as the body ratio, wherein h up =p s ·y−p h ·y and h low =p h ·y−p k ·y, p s  is a middle position between the positions of two shoulder key points, p s ·y is a y coordinate of p s , p h ·y is a y coordinate of p h , p k  is a middle position between the positions of two knee key points, and pk·y is a y coordinate of p k ; 
 obtaining an angle between a vector {right arrow over (v)} hs  between the middle position p h  and the middle position p s , and a unit vector {right arrow over (y)} in y-direction to take as the upper body angle; and 
 the determining the posture of the human based on the position of the intersection of the body ratio and the upper body angle in the second feature map comprises: 
 determining the posture of the human according to the position of the intersection of the body ratio and the upper body angle in the second feature map with respect to the first threshold curve for distinguishing the standing posture and the sitting posture and the second threshold curve for distinguishing the lying posture, the standing posture, and the sitting posture. 
 
     
     
       14. The mobile machine of  claim 13 , wherein the one or more programs further include instructions to:
 in response to the detected key points including one shoulder key point, obtain the position of another of the two shoulder key points based on the position of the detected shoulder key point; 
 in response to the detected key points including one hip key point, obtain the position of another of the two hip key points based on the position of the detected hip key point; and 
 in response to the detected key points including one knee key point, obtain the position of another of the two knee key points based on the position of the detected knee key point. 
 
     
     
       15. The mobile machine of  claim 9 , wherein the one or more programs further include instructions to:
 in response to the detected key points including a plurality of head key points among the predefined key points, calculate a head height H head  based on the head key points; and 
 determine the posture of the human by comparing the head height H head  with a first head height threshold for distinguishing a standing posture and a sitting posture and a second head height threshold for distinguishing the sitting posture and a lying posture. 
 
     
     
       16. The mobile machine of  claim 9 , wherein the predefined key points include two eye key points, a nose key point, two ear key points, a neck key point, two shoulder key points, two elbow key points, two hand key points, two hip key points, two knee key points, and two foot key points, and wherein the two eye key points, the nose key point, and the two ear key points are referred to as head key points. 
     
     
       17. The method of  claim 8 , further comprising:
 determining whether the detected key points comprise all the predefined key points, when the determined posture is not the lying posture; 
 in response to the detected key points not comprising all the predefined key points, determining whether or not the detected key points comprising at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points; 
 in response to the detected key points comprising at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points, obtaining a detected body ratio of the body of the human based on the positions of the detected key points; and 
 determining the posture of the human based on a position of an intersection of the detected body ratio in the first feature map. 
 
     
     
       18. The method of  claim 17 , further comprising:
 determining whether or not the detected key points comprise a plurality of the head key points among the predefined key points, in response to the detected key points not comprising all the predefined key points, and the detected key points not comprising at least a shoulder key point, at least a hip key point, and at least a knee key point among the predefined key points; 
 in response to the detected key points comprising the plurality of the head key points, calculating a head height based on the plurality of the head key points; 
 comparing the head height with a first head height threshold for distinguishing the standing posture and the sitting posture, and a second head height threshold for distinguishing the sitting posture and the lying posture; 
 in response to the head height being larger than the first head height threshold, determining that the posture of the human is the standing posture; 
 in response to the head height being between the first head height threshold and the second head height threshold, determining that the posture of the human is the sitting posture; and 
 in response to the head height being smaller than the second head height threshold, determining that the posture of the human is the lying posture. 
 
     
     
       19. The method of  claim 1 , wherein each of the predefined feature maps comprises values corresponding to features for distinguishing postures and is used to classify the detected key points, and postures of the human in each of the predefined feature maps are separated using a maximum margin from each posture region in each of the predefined feature maps. 
     
     
       20. The method of  claim 1 , wherein feature values shown in the first feature map and the second feature map are normalized values with mean and scale.