Method and system for adjusting audio signals based on motion deviation

Abstract

A system measures performance activity of a user using a motion capture device, such as one or more knee sleeves each a plurality of inertial-measurement unit. Captured motion characteristic data is classified against previously captured data and then mapped to particular audio effects, which are then altered in different ways depending on the classification and provided the user. The audio feedback works to improve the user's performance of the activity by changing between negative and positive feedback depending on the user's performance. The system regularly measures the motion capture data, altering audio effects provided to the user, until the user reaches an improvement goal.

What is claimed: 
     
       1. A system to modify an audio signal in response to changes in motion patterns of a wearer of a leg prosthesis, the system comprising:
 the leg prosthesis comprising a socket portion and a lower portion connected by a knee joint, the leg prosthesis further comprising at least one sensor; 
 a memory associating a plurality of performance activities with a plurality of motion characteristics and a plurality of controllable audio features, wherein a particular performance activity of the plurality of performance activities is associated with a particular set of motion characteristics of the plurality of motion characteristics and a particular set of controllable audio features of the plurality of controllable audio features; and 
 one or more processors configured to:
 receive the audio signal from a source, wherein the source stores the audio signal, 
 identify a performance of a performance activity for assessment, 
 consult the memory to identify a first set of motion characteristics and a first set of controllable audio features associated with the performance of the performance activity, 
 receive sensor motion data from the at least one sensor, 
 determine a plurality of measurements for the first set of motion characteristics based on the sensor motion data; 
 determine a deviation value and deviation type based on differences between the plurality of measurements and a corresponding plurality of user-specific target measurements, wherein the corresponding plurality of user-specific target measurements are generated based at least in part on measurements associated with a prior performance of the performance activity by the wearer; 
 based at least in part on a determination that the deviation value is greater than a first deviation threshold and less than a second deviation threshold, modify the received audio signal by adjusting at least one controllable audio feature of the first set of controllable audio features by a first amount, wherein the at least one controllable audio feature is based at least in part on the deviation type; and 
 based at least in part on a determination that deviation value is greater than the second deviation threshold, modify the received audio signal by adjusting the at least one controllable audio feature of the first set of controllable audio features by a second amount, and wherein the second deviation threshold is greater than the first deviation threshold the second amount is greater than the first amount. 
 
 
     
     
       2. The system of  claim 1 , wherein the at least one controllable audio feature of the first set of controllable audio features corresponds to at least one of amplitude modulation, frequency modulation, distortion, bitcrushing, chorus, ducking, frequency filtering, spatialization, localization, panning, reverb, echo, compression, expanding, phasing, pitch-shifting, flanging, gating, vocoding, or delay. 
     
     
       3. The system of  claim 2 , wherein the frequency filtering corresponds to at least one of a high-pass filtering, a bass-cut filtering, a band-pass filtering, a low-shelf filtering, a high-shelf filtering, or a band-stop filtering. 
     
     
       4. The system of  claim 1 , wherein the audio signal includes music, wherein the modified audio signal becomes increasingly unmodified relative to the received audio signal as the deviation value approaches zero. 
     
     
       5. The system of  claim 1 , wherein the memory associates only one motion characteristic with each of the plurality of controllable audio features. 
     
     
       6. The system of  claim 1 , wherein the memory associates more than one motion characteristic with each of the plurality of controllable audio features. 
     
     
       7. The system of  claim 1 , wherein the at least one controllable audio feature of the first set of controllable audio features comprises at least two controllable audio features. 
     
     
       8. The system of  claim 1 , wherein the first set of motion characteristics comprises at least two different motion characteristics. 
     
     
       9. The system of  claim 1 , wherein the first set of motion characteristics comprises at least three different motion characteristics. 
     
     
       10. The system of  claim 1 , wherein the first set of motion characteristics comprises at least one of gait speed, step width, step length, stance time, gait phase, symmetry of limbs, inertial motions of the limbs, cadence, path of pressure, path of center of mass, ground reaction force, lower limb trajectory, trunk orientation, trunk flexion, trunk extension, hip flexion, hip extension, knee flexion, knee extension, foot-floor angle at heel contact, ankle plantarflexion, ankle dorsiflexion, or energy expenditure. 
     
     
       11. The system of  claim 1 , wherein the plurality of measurements is a second plurality of measurements, wherein the sensor motion data is second sensor motion data, wherein the performance of the performance activity is a second performance of the performance activity, wherein the prior performance of the performance activity is a first performance of the performance activity, wherein the measurements associated with a prior performance of the performance activity is a first plurality of measurements, and wherein the one or more processors are configured to:
 receive first sensor motion data associated with the first performance of the performance activity by the wearer; 
 determine the first plurality of measurements for the first set of motion characteristics based on the first sensor motion data, and 
 generate the first deviation threshold and the second deviation threshold using the first plurality of measurements, wherein the first deviation threshold and the second deviation threshold correspond to the first performance of the performance activity by the wearer. 
 
     
     
       12. The system of  claim 1 , wherein the performance of the performance activity is a second performance of the performance activity by the wearer, wherein the prior performance of the performance activity is a first performance of the performance activity, and wherein the one or more processors are configured to:
 generate the first deviation threshold and the second deviation threshold using the plurality of prior measurements associated with the first performance of the performance activity by the wearer. 
 
     
     
       13. The system of  claim 1 ,
 wherein based at least in part on a determination that the deviation value exceeds at least the first deviation threshold, the one or more processors modify the received audio signal to provide negative audio feedback, wherein the negative audio feedback corresponds to a distortion of the received audio signal, and 
 wherein based at least in part on a determination that the deviation value does not exceed the first deviation threshold, the one or more processors modify the received audio signal by a third amount to provide positive audio feedback, wherein the positive audio feedback includes at least one of a bass boost or increase in tempo. 
 
     
     
       14. The system of  claim 1 , wherein the plurality of measurements is automatically classified into a performance category, and wherein each performance category corresponds to at least one controllable audio feature of the first set of controllable audio features. 
     
     
       15. The system of  claim 1 , wherein the one or more processors are further configured to determine the deviation type by using an artificial intelligence to compare the plurality of measurements and the corresponding plurality of user-specific target measurements. 
     
     
       16. The system of  claim 1 , wherein the one or more processors are further configured to determine the deviation value based on weighted differences between the plurality of measurements and the corresponding plurality of user-specific target measurements. 
     
     
       17. The system of  claim 1 , wherein the at least one sensor is a first sensor, and wherein the system further comprises a second sensor on a sound limb of the wearer, wherein the sensor motion data is from the first sensor and the second sensor. 
     
     
       18. A computer-implemented method to modify an audio signal that is to be provided to a subject in response to changes in motion patterns of the subject, the method comprising:
 identifying a first performance activity for assessment; 
 consulting a memory to identify a first set of motion characteristics and a first set of controllable audio features associated with the first performance activity, wherein the memory associates a plurality of performance activities with a plurality of motion characteristics and a plurality of controllable audio features, wherein a particular performance activity of the plurality of performance activities is associated with a particular set of motion characteristics of the plurality of motion characteristics and a particular set of controllable audio features of the plurality of controllable audio features, wherein the first set of motion characteristics comprises multiple motion characteristics of the plurality of motion characteristics; 
 receiving the audio signal from a source; 
 receiving sensor motion data from at least one sensor coupled to a leg prosthesis worn by the subject; 
 determining a plurality of measurements for the first set of motion characteristics based on the sensor motion data; 
 determine a deviation value and a deviation type by comparing the plurality of measurements to corresponding baseline measurements and target measurements wherein the baseline measurements and the target measurements are different and are generated based at least in part on measurements associated with a prior performance of the first performance activity by the subject; 
 based at least in part on a determination that the deviation value is greater than a first deviation threshold and less than a second deviation threshold, modifying the audio signal by adjusting at least one controllable audio feature of the first set of controllable audio features of the audio signal by a first amount, wherein the first deviation threshold is based at least in part on the target measurements, and the second deviation threshold is greater than the first deviation threshold and is based at least in part on the baseline measurements, and wherein the at least one controllable audio feature is based at least in part on the deviation type; and 
 based at least in part on a determination that the deviation value is greater than the second deviation threshold, modify the received audio signal by adjusting the at least one controllable audio feature of the first set of controllable audio features by a second amount, wherein the second amount is greater than the first amount. 
 
     
     
       19. The method of  claim 18 , wherein the at least one controllable audio feature of the first set of controllable audio features corresponds to at least one of amplitude modulation, frequency modulation, distortion, bitcrushing, chorus, ducking, frequency filtering, spatialization, localization, panning, reverb, echo, compression, phasing, flanging, gating, vocoding, or delay on the audio signal. 
     
     
       20. The method of  claim 19 , wherein the frequency filtering corresponds to at least one of a high-pass filtering, bass-cut filtering, a band-pass filtering, a low-shelf filtering, a high-shelf filtering, or a band-stop filtering. 
     
     
       21. The method of  claim 18 , wherein the audio signal includes music, wherein the modified audio signal approaches the received audio signal as the deviation value approaches zero. 
     
     
       22. The method of  claim 18 , wherein the first set of motion characteristics comprises at least one of gait speed, step width, step length, stance time, gait phase, symmetry of limbs, inertial motions of the limbs, cadence, path of pressure, path of center of mass, ground reaction force, lower limb or sacral trajectory, or trunk orientation. 
     
     
       23. The method of  claim 18 , wherein the first set of motion characteristics comprises at least one trunk flexion, trunk extension, hip flexion, hip extension, knee flexion, knee extension, foot-floor angle at heel contact, ankle plantarflexion, ankle dorsiflexion, or energy expenditure. 
     
     
       24. The method of  claim 18 , wherein the at least one controllable audio feature of the first set of controllable audio features comprises at least two controllable audio features. 
     
     
       25. The method of  claim 18 , wherein the first set of motion characteristics comprises at least two different motion characteristics. 
     
     
       26. The method of  claim 18 , wherein the plurality of measurements is automatically classified into a performance category, wherein the performance category corresponds to at least one of sub-par performance or acceptable performance, and wherein each performance category corresponds to at least one controllable audio feature of the first set of controllable audio features. 
     
     
       27. The method of  claim 18 , wherein the deviation type is determined by using an artificial intelligence to compare the plurality of measurements and the corresponding baseline measurements and the target measurements. 
     
     
       28. The method of  claim 18 , wherein the at least one sensor is a first sensor, and wherein the sensor motion data is received from the first sensor and a second sensor on a sound limb of the subject. 
     
     
       29. A non-transitory computer-readable storage medium storing computer-executable instructions that when executed by one or more processors cause the one or more processors to:
 identify a first performance activity for assessment; 
 consult a memory to identify a first set of motion characteristics and a first set of controllable audio features associated with the first performance activity, wherein the memory associates a plurality of performance activities with a plurality of motion characteristics and a plurality of controllable audio features, wherein a particular performance activity of the plurality of performance activities is associated with a particular set of motion characteristics of the plurality of motion characteristics and a particular set of controllable audio features of the plurality of controllable audio features, wherein the first set of motion characteristics comprises multiple motion characteristics of the plurality of motion characteristics; 
 receive an audio signal from a source; 
 receive sensor motion data that corresponds to a subject; 
 determine a plurality of measurements for the first set of motion characteristics based on the sensor motion data; 
 determine a deviation value and deviation type based on differences between the plurality of measurements and corresponding subject-specific target measurements and subject-specific baseline measurements, wherein the corresponding subject-specific target measurements are generated based at least in part on measurements associated with a prior performance of the performance activity by the subject; 
 based at least in part on a determination that the deviation value is greater than a first deviation threshold that corresponds to the subject-specific target measurements and less than a second deviation threshold that is greater than the first deviation threshold and that corresponds to the subject-specific baseline measurements, modify the audio signal by adjusting at least one controllable audio feature of the first set of controllable audio features of the audio signal by a first amount, wherein the at least one controllable audio feature is based at least in part on the deviation type; and 
 based at least in part on a determination that the deviation value is greater than the second deviation threshold, modify the received audio signal by adjusting the at least one controllable audio feature of the first set of controllable audio features by a second amount that is greater than the first amount. 
 
     
     
       30. The non-transitory computer-readable storage medium of  claim 29 , wherein the sensor motion data corresponds to data received from a sensor coupled to a leg prosthesis worn by the subject. 
     
     
       31. The non-transitory computer-readable storage medium of  claim 29 , wherein the plurality of measurements is automatically classified into a performance category, and wherein each performance category corresponds to at least one controllable audio feature of the first set of controllable audio features. 
     
     
       32. The non-transitory computer-readable storage medium of  claim 29 , the deviation value is determined based on weighted differences between the plurality of measurements and the corresponding subject-specific target measurements and the subject-specific baseline measurements.