Hand-motion monitoring apparatus and hand-motion monitoring method

Abstract

Included are a behavior recognition unit that recognizes from the transition state of the posture of a subject person a hand-motion behavior that is a linkage of a plurality of local actions based on pieces of three-dimensional skeleton information sequentially obtained by a skeleton information acquisition unit, a behavior checking unit that checks the hand-motion behavior recognized by the behavior recognition unit while determining if the recognition accuracy of each of the local actions of the hand-motion behavior is lower than a predetermined threshold, and an action improvement proposal unit that, based on the result of the check performed by the behavior checking unit, gives the subject person feedback about a proposal for improving a local action with recognition accuracy lower than the threshold among the local actions of the hand-motion behavior.

1 . A hand-motion monitoring apparatus comprising:
 an imaging unit that sequentially obtains RGB images and depth images by imaging an upper body including both hands of a subject person;   an image coordinates setting unit that sets, as image coordinates, a plurality of representative skeleton points provided to the upper body mainly including the both hands of the subject person while estimating a posture of the upper body of the subject person based on the RGB images sequentially obtained from the imaging unit;   a coordinates extraction unit that sequentially extracts, from the image coordinates set by the image coordinates setting unit image, coordinates related to a hand-motion behavior of the subject person based on a transition state of the posture of the subject person;   a skeleton information acquisition unit that obtains pieces of temporally continuous three-dimensional skeleton information on mainly the both hands of the subject person by temporally synchronizing the image coordinates sequentially extracted by the coordinates extraction unit with the depth images sequentially obtained from the imaging unit;   a behavior recognition unit that recognizes, from the transition state of the posture of the subject person, a hand-motion behavior that is a linkage of a plurality of local actions based on the pieces of three-dimensional skeleton information sequentially obtained by the skeleton information acquisition unit;   a behavior checking unit that checks the hand-motion behavior recognized by the behavior recognition unit while determining if recognition accuracy of each of the local actions of the hand-motion behavior is lower than a predetermined threshold; and   an action improvement proposal unit that, based on a result of the check performed by the behavior checking unit, gives the subject person feedback about a proposal for improving a local action with recognition accuracy lower than the threshold among the local actions of the hand-motion behavior.   
     
     
         2 . The hand-motion monitoring apparatus according to  claim 1 , wherein
 the image coordinates setting unit sets the image coordinates on a two-dimensional coordinate system including one of right and left shoulder portions of the subject person as a coordinate origin, an X-axis lying in a direction of a line connecting the right and left shoulder portions with reference to the coordinate origin, and a Z-axis lying in a vertical direction with respect to the X-axis, and   the skeleton information acquisition unit forms a three-dimensional coordinate system by combining the depth images sequentially obtained from the imaging unit with the two-dimensional coordinate system of the image coordinates sequentially extracted by the coordinates extraction unit while setting a depth direction of the depth images as a Y-axis, and thus obtains the three-dimensional skeleton information.   
     
     
         3 . The hand-motion monitoring apparatus according to  claim 1 , wherein the behavior recognition unit sequentially recognizes the local actions from the pieces of three-dimensional skeleton information sequentially obtained by the skeleton information acquisition unit while referring to a hand-motion-action recognition model that has been constructed through deep learning by using as training data an action recognition pattern set for each local action. 
     
     
         4 . The hand-motion monitoring apparatus according to  claim 1 , wherein
 the behavior checking unit sets in advance a threshold for determining a shortage of action time of each local action so as to determine if each local action has terminated within the corresponding threshold, and confirms that a local action that has terminated within the corresponding threshold is recognized erroneously, and   the action improvement proposal unit gives feedback about a proposal for improving the local action that has been confirmed to be recognized erroneously by the behavior checking unit.   
     
     
         5 . The hand-motion monitoring apparatus according to  claim 1 , wherein regarding a local action that has not been recognized by the behavior recognition unit due to a lack of the three-dimensional skeleton information among the local actions of the hand-motion behavior, the action improvement proposal unit gives feedback to the effect that the local action is not recognizable. 
     
     
         6 . A hand-motion monitoring method comprising:
 a first step of sequentially obtaining RGB images and depth images by imaging an upper body including both hands of a subject person;   a second step of setting, as image coordinates, a plurality of representative skeleton points provided to the upper body mainly including the both hands of the subject person while estimating a posture of the upper body of the subject person based on the RGB images sequentially obtained in the first step;   a third step of sequentially extracting from the image coordinates set in the second step image coordinates related to a hand-motion behavior of the subject person based on a transition state of the posture of the subject person;   a fourth step of obtaining pieces of temporally continuous three-dimensional skeleton information on mainly the both hands of the subject person by temporally synchronizing the image coordinates sequentially extracted in the third step with the depth images sequentially obtained in the first step;   a fifth step of recognizing, from the transition state of the posture of the subject person, a hand-motion behavior that is a linkage of a plurality of local actions based on the pieces of three-dimensional skeleton information sequentially obtained in the fourth step;   a sixth step of checking the hand-motion behavior recognized in the fifth step while determining if recognition accuracy of each of the local actions of the hand-motion behavior is lower than a predetermined threshold; and   a seventh step of, based on a result of the check performed in the sixth step, giving the subject person feedback about a proposal for improving a local action with recognition accuracy lower than the threshold among the local actions of the hand-motion behavior.   
     
     
         7 . The hand-motion monitoring method according to  claim 6 , wherein
 the second step includes setting the image coordinates on a two-dimensional coordinate system including one of right and left shoulder portions of the subject person as a coordinate origin, an X-axis lying in a direction of a line connecting the right and left shoulder portions with reference to the coordinate origin, and a Z-axis lying in a vertical direction with respect to the X-axis, and   the fourth step includes forming a three-dimensional coordinate system by combining the depth images sequentially obtained in the first step with the two-dimensional coordinate system of the image coordinates sequentially extracted in the third step while setting a depth direction of the depth images as a Y-axis, and thus obtaining the three-dimensional skeleton information.   
     
     
         8 . The hand-motion monitoring method according to  claim 6 , wherein the fifth step includes sequentially recognizing the local actions from the pieces of three-dimensional skeleton information sequentially obtained in the fourth step while referring to a hand-motion-action recognition model that has been constructed through deep learning by using as training data an action recognition pattern set for each local action. 
     
     
         9 . The hand-motion monitoring method according to  claim 6 , wherein
 the sixth step includes setting in advance a threshold for determining a shortage of action time of each local action so as to determine if each local action has terminated within the corresponding threshold, and confirms that a local action that has terminated within the corresponding threshold is recognized erroneously, and   the seventh step includes, regarding the local action that has been confirmed to be recognized erroneously in the sixth step, giving feedback about a proposal for improving the local action.   
     
     
         10 . The hand-motion monitoring method according to  claim 6 , wherein the seventh step includes, regarding a local action that has not been recognized in the fifth step due to a lack of the three-dimensional skeleton information among the local actions of the hand-motion behavior, giving feedback to the effect that the local action is not recognizable.