System for correcting user movement

Abstract

Characteristics of a user's movement are evaluated based on performance of activities by a user within a field of view of a camera. Video data representing performance of a series of movements by the user is acquired by the camera. Pose data is determined based on the video data, the pose data representing positions of the user's body while performing the movements. The pose data is compared to a set of existing videos that correspond to known errors to identify errors performed by the user. The errors may be used to generate scores for various characteristics of the user's movement. Based on the errors, exercises or other activities to improve the movement of the user may be determined and included in an output presented to the user.

What is claimed is: 
     
       1. A system comprising:
 one or more memories storing computer-executable instructions; and 
 one or more hardware processors to execute the computer-executable instructions to:
 cause a user device to present instructions to perform a plurality of movements; 
 acquire, using a camera of the user device, video data that represents a user within a field of view of the camera performing the plurality of movements; 
 determine pose data based on the video data, wherein the pose data is indicative of one or more positions of the user during performance of the plurality of movements; 
 determine correspondence between the pose data and movement data, wherein the movement data associates poses of users with errors in movement of the users; 
 determine, based on the correspondence between the pose data and the movement data, one or more errors associated with the pose data; 
 determine correspondence between the one or more errors and score data, wherein the score data associates errors with score values; 
 determine, based on the correspondence between the one or more errors and the score data, one or more score values associated with the pose data; 
 determine, based on the one or more score values, at least one score indicative of a characteristic of movement of the user; 
 determine correspondence between the one or more errors and activity data, wherein the activity data associates errors with activities to improve movements; 
 determine, based on the correspondence between the one or more errors and the activity data, at least one activity to improve the characteristic of movement of the user; and 
 cause the user device to present an output indicative of the at least one score and the at least one activity. 
 
 
     
     
       2. The system of  claim 1 , further comprising computer-executable instructions to:
 determine correspondence between the pose data and segmentation data indicative of at least a first movement and a second movement of the plurality of movements; and 
 determine, based on the correspondence between the pose data and the segmentation data: a first portion of the pose data associated with the first movement by the user and a second portion of the pose data associated with the second movement by the user; 
 wherein at least a subset of the one or more errors is associated with the first movement. 
 
     
     
       3. The system of  claim 1 , wherein the movement data includes one or more of a plurality of video data or a plurality of pose data, and the computer-executable instructions to determine the correspondence between the pose data and the movement data include computer-executable instructions to:
 use a machine learning system to classify the pose data using the one or more of the plurality of video data or the plurality of pose data, wherein the one or more errors are determined based on errors associated with at least a subset of the one or more of the plurality of video data or the plurality of pose data. 
 
     
     
       4. A method comprising:
 determining movement data that associates each position of a plurality of positions with a respective corresponding error data; 
 determining first video data that represents a user performing one or more movements; 
 determining pose data based on the first video data, wherein the pose data is indicative of one or more positions of the user; 
 determining a first position of the one or more positions based on the pose data; 
 determining a first error associated with the first position of the user based on correspondence between the first position and a second position indicated in the movement data, wherein the movement data associates the second position with the first error; 
 determining a first score value based on the first error; and 
 determining a first output indicative of the first score value. 
 
     
     
       5. The method of  claim 4 , further comprising:
 determining correspondence between activity data and one or more of the first position or the first error, wherein the activity data associates the one or more of the first position or the first error with at least one activity; and 
 determining a second output indicative of the at least one activity. 
 
     
     
       6. The method of  claim 4 , further comprising:
 determining a second error based on correspondence between the pose data and the movement data, wherein the first error is associated with a first portion of a body of the user and the second error is associated with a second portion of the body of the user; 
 determining a second score value based on the second error; and 
 including, in the first output, an indication of the first score value in association with an indication of the first portion of the body and an indication of the second score value in association with an indication of the second portion of the body. 
 
     
     
       7. The method of  claim 4 , further comprising:
 determining a second error based on correspondence between the pose data and the movement data; 
 determining a second score value based on the second error; 
 determining a first weight value associated with the first score value and a second weight value associated with the second score value; 
 determining a third score value based on the first weight value, the first score value, the second weight value, and the second score value; and 
 including the third score value in the first output. 
 
     
     
       8. The method of  claim 4 , wherein the first score value is associated with a first characteristic of movement of the user, the method further comprising:
 determining a second score value based on the first error, wherein the second score value is associated with a second characteristic of the movement of the user; and 
 including an indication of the second score value in the first output. 
 
     
     
       9. The method of  claim 4 , further comprising:
 determining, based on correspondence between the pose data and movement data, a first severity level associated with the first error, wherein the first severity level is associated with the first score value and a second severity level for the first error is associated with a second score value. 
 
     
     
       10. The method of  claim 4 , further comprising:
 determining a second error based on correspondence between the pose data and the movement data; 
 determining, based on the movement data, a first priority value associated with the first error and a second priority value associated with the second error; 
 determining that the first priority value indicates a greater priority than the second priority value; 
 based on the first priority value indicating the greater priority than the second priority value, determining correspondence between the first error and activity data, wherein the activity data associates the first error with at least one activity; and 
 including an indication of the at least one activity in the first output. 
 
     
     
       11. The method of  claim 4 , wherein the first score value is associated with the user performing the one or more movements at a first time, the method further comprising:
 determining a second score value associated with the user performing the one or more movements at a second time; and 
 including, in the first output, an indication of the first score value in association with the first time and an indication of the second score value in association with the second time. 
 
     
     
       12. The method of  claim 4 , further comprising:
 before determining the pose data, determining, based on one or more of the first video data or second video data, that at least a threshold portion of a body of the user is within a field of view of a camera. 
 
     
     
       13. A system comprising:
 one or more memories storing computer-executable instructions; and 
 one or more hardware processors to execute the computer-executable instructions to:
 determine movement data that associates each position of a plurality of positions with a respective corresponding error data; 
 determine first video data that represents a user performing one or more movements; 
 determine first pose data based on the first video data, wherein the first pose data is indicative of one or more positions of the user; 
 determine a first position of the user based on the pose data; 
 determine a first error associated with the first position of the user based on correspondence between the first position and a second position indicated in the movement data, wherein the movement data associates the second position with the first error, and wherein the first error is associated with a characteristic of the one or more movements; 
 determine at least one first activity associated with the characteristic based on correspondence between one or more of the first position or the first error and activity data that associates the one or more of the first position or the first error with the at least one first activity; and 
 determine a first output indicative of the at least one first activity. 
 
 
     
     
       14. The system of  claim 13 , further comprising computer-executable instructions to:
 determine correspondence between the first error and score data, wherein the score data associates the first error with a first score value; and 
 determine a second output indicative of the first score value. 
 
     
     
       15. The system of  claim 13 , further comprising computer-executable instructions to:
 determine, based on correspondence between the first pose data and the movement data, one or more positions that are constrained from performance by the user; 
 determine, based on the activity data, a portion of the at least one first activity associated with the one or more positions; and 
 include, in the first output, an indication of one or more modifications to the at least one first activity based on the one or more positions. 
 
     
     
       16. The system of  claim 13 , further comprising computer-executable instructions to:
 at a first time, determine correspondence between the first error and score data, wherein the score data associates the first error with a first score value; 
 determine an indication of performance of the at least one first activity by the user; 
 at a second time after the first time, determine second video data that represents the user performing the one or more movements; 
 determine second pose data based on the second video data; 
 determine a second error based on correspondence between the second pose data and the movement data; 
 determine a second score value based on correspondence between the second error and the score data; and 
 determine at least one second activity based on the activity data and a difference between the second score value and the first score value. 
 
     
     
       17. The system of  claim 13 , further comprising computer-executable instructions to:
 at a first time, determine an indication of performance of the at least one first activity by the user; 
 at a second time after the first time, determine second video data that represents the user performing the one or more movements; 
 determine second pose data based on the second video data; 
 determine the first error based on correspondence between the second pose data and the movement data; 
 based on the indication of the performance of the at least one first activity by the user, determine at least one second activity based on correspondence between the first error and the activity data; and 
 determine a second output indicative of the at least one second activity. 
 
     
     
       18. The system of  claim 13 , wherein the movement data includes one or more of second video data or second pose data associated with an indication of the first error, and the computer-executable instructions to determine the first error based on the correspondence between the first position and the second position indicated in the movement data further comprise computer-executable instructions to:
 use a machine learning system to classify the first position as corresponding to the second position included in the one or more of the second video data or the second pose data within at least a threshold confidence value. 
 
     
     
       19. The system of  claim 13 , further comprising computer-executable instructions to:
 determine correspondence between the first pose data and segmentation data indicative of a first movement of the one or more movements; and 
 determine, based on the correspondence between the first pose data and the segmentation data, a first portion of the first pose data associated with the first movement; 
 wherein the first error is associated with the first movement. 
 
     
     
       20. The system of  claim 13 , further comprising computer-executable instructions to:
 determine, based on one or more of the first video data or second video data, that at least a threshold portion of a body of the user is within a field of view of a camera.