Computer vision-based assessment of posture of living bodies and approaches to surfacing insights gained therefrom

Abstract

Introduced here are computer-implemented platforms (also referred to as “therapy platforms”) that are designed to provide feedback for posture improvement. As part of a posture program, a user may be requested to engage with a therapy platform, which can capture image data of a user in an environment over a time period (e.g., a posture session) to analyze for posture feedback. Simultaneously to capturing the image data, the therapy platform can extract an estimated pose of a user based on the image data and determine a posture indicator based on the estimated pose. The therapy platform uses the posture indicator to render feedback and other content (e.g., visual instructions) for the user for improving her posture.

What is claimed is: 
     
         1 . A method performed by a computer program executing on a computing device, the method comprising:
 capturing video data over a time period, the video data depicting a user in an environment;   simultaneously to capturing the video data over the time period:   extracting an estimated pose of the user based on the video data;   determining a posture indicator by heuristically analyzing the estimated pose; and   rendering, via a display on the computing device, feedback for the user based on the posture indicator.   
     
     
         2 . The method of  claim 1 , wherein the feedback is sent to a second computing device of a physical therapist, coach, or doctor. 
     
     
         3 . The method of  claim 1 , further comprising:
 rendering, via the display of the computing device, instructions to position the computing device in the environment such that a minimum number of the user's joints are depicted in the video data.   
     
     
         4 . The method of  claim 1 , wherein the posture indicator is a forward gaze, relaxed arms, or posture change. 
     
     
         5 . The method of  claim 4 , wherein:
 forward gaze comprises neck angle between neck-hip and neck-headtop bones;   relaxed arms comprises arm angle between shoulder-hip and shoulder-elbow bones; and   posture change comprises backbone direction given by shoulder-hip bone.   
     
     
         6 . The method of  claim 1 , wherein extracting the estimated pose of the user based on the video data comprises:
 extracting a feature map from the video data; and   applying a pose engine to the feature map.   
     
     
         7 . The method of  claim 6 , wherein the pose engine based on a neural network. 
     
     
         8 . The method of  claim 1 , further comprising:
 computing deviations between the estimated pose and a neutral position;   adding, to a tally of deviations, each significant deviation of the deviations; and   determining the feedback based on the tally.   
     
     
         9 . The method of  claim 8 , wherein significant deviations are determined by:
 for each deviation:
 responsive to determining a value of the deviation is greater than a threshold and determining a time period associated with the deviation is greater than a threshold amount of time, determining that the deviation is a significant deviation. 
   
     
     
         10 . A method performed by a computer program executing on a computing device, the method comprising:
 capturing image data over a time period, the image data depicting a user in an environment;   extracting an estimated pose of the user based on the image data;   determining a posture indicator by heuristically analyzing the estimated pose; and   rendering, via a display on the computing device, feedback for the user based on the posture indicator.   
     
     
         11 . The method of  claim 10 , wherein the feedback is sent to a second computing device of a physical therapist, coach, or doctor. 
     
     
         12 . The method of  claim 10 , further comprising:
 rendering, via the display of the computing device, instructions to position the computing device in the environment such that a minimum number of the user's joints are depicted in the image data.   
     
     
         13 . The method of  claim 10 , wherein the posture indicator is a forward gaze, relaxed arms, or posture change. 
     
     
         14 . The method of  claim 13 , wherein:
 forward gaze comprises neck angle between neck-hip and neck-headtop bones;   relaxed arms comprises arm angle between shoulder-hip and shoulder-elbow bones; and   posture change comprises backbone direction given by shoulder-hip bone.   
     
     
         15 . The method of  claim 10 , wherein extracting the estimated pose of the user based on the image data comprises:
 extracting a feature map from the image data; and   applying a neural network to the feature map.   
     
     
         16 . The method of  claim 10 , further comprising:
 computing deviations between the estimated pose and a neutral position;   adding, to a tally of deviations, each significant deviation of the deviations; and   determining the feedback based on the tally.   
     
     
         17 . The method of  claim 16 , wherein significant deviations are determined by:
 for each deviation:   responsive to determining a value of the deviation is greater than a threshold and determining a time period associated with the deviation is greater than a threshold amount of time, determining that the deviation is a significant deviation.   
     
     
         18 . A computing device comprising:
 a display;   a processor; and   a memory storing instructions that, when executed by the processor, cause the processor to perform operations comprising:
 capturing video data over a time period, the video data depicting a user in an environment; 
 simultaneously to capturing the video data over the time period: 
 extracting an estimated pose of the user based on the video data; 
 determining a posture indicator by heuristically analyzing the estimated pose; and 
 rendering, via the display, feedback for the user based on the posture indicator. 
   
     
     
         19 . The computing device of  claim 18 , wherein the feedback is sent to a second computing device of a physical therapist, coach, or doctor. 
     
     
         20 . The computing device of  claim 18 , wherein the operations further comprise:
 rendering, via the display of the computing device, instructions to position the computing device in the environment such that a minimum number of the user's joints are depicted in the video data.   
     
     
         21 . The computing device of  claim 18 , wherein the posture indicator is a forward gaze, relaxed arms, or posture change.