Co-adaptation for learning and control of devices

Abstract

A method of operating a biological interface is disclosed. The method may include obtaining an input physiological or neural signal from a subject, acquiring an input set of values from the input signal, obtaining a predictive signal from the subject or the environment, acquiring a predictive set of values from the predictive signal, training a decoder function in response to data from the predictive set of values, performing at least one calculation on the input set of values using the decoder function to produce an output set of values, and operating a device with the output set of values. A biological interface system is also disclosed. The biological interface system may contain an input signal sensor, an input signal processor, a predictive signal processor, a memory device storing data, and a system processor coupled to the memory device and configured to execute a decoder function.

What is claimed is: 
     
       1. A method of operating a biological interface device, the method comprising:
 obtaining an input signal from a subject, the input signal associated with a command for operation of the biological interface device and comprising a neural signal or a physiological signal; 
 processing the input signal to acquire an input set of values; 
 obtaining a predictive signal from the subject or an environment of the subject, the predictive signal associated with an intent for operation of the biological interface device and comprising at least one physiological predictive signal measuring physical movement of the subject; 
 processing the predictive signal to acquire a predictive set of values; 
 performing at least one calculation on the input set of values and the predictive set of values using a decoder function trained with historical data from historical input signals and historical predictive signals to produce an output set of values, the historical data including a matrix F of features and a matrix K of desired kinematic control signals for providing the output set of values, such that the matrix K is constructed by matrix multiplication of F by a matrix of regression coefficients C which represents weighted contribution of the features; 
 operating the biological interface device with the output set of values; 
 obtaining a feedback signal associated with the operation of the biological interface device and comprising a movement signal of the biological interface device; 
 processing the feedback signal to acquire a feedback set of values; 
 determining a difference between the feedback set of values and the predictive set of values; and 
 re-training the decoder function if the difference between the feedback set of values and the predictive set of values is greater than a predetermined threshold, thereby performing co-adaptation to incorporate the feedback signal to iteratively improve the decoder function while the subject is attempting control via the biological interface device. 
 
     
     
       2. The method of  claim 1 , wherein the predictive signal comprises at least one of the physiological predictive signal and at least one environmental predictive signal. 
     
     
       3. The method of  claim 2 , wherein processing the predictive signal to acquire the predictive set of values comprises:
 performing at least one calculation on data from the predictive signal using a predictive decoder function trained with historical data from historical predictive signals to produce the predictive set of values. 
 
     
     
       4. The method of  claim 2 , wherein the environmental predictive signal comprises at least one of an orientation signal, location signal, surroundings signal, temporal signal, sound signal, visual signal, verbal command, and visual command. 
     
     
       5. The method of  claim 2 , wherein the environmental predictive signal is associated with a software log of actions. 
     
     
       6. The method of  claim 1 , wherein the physiological predictive signal comprises at least one of an eye tracking signal, voice signal, movement signal, cardiac signal, skin conductance signal, and muscular signal. 
     
     
       7. The method of  claim 1 , wherein the movement signal is associated with movement of the biological interface device or movement of a cursor associated with the biological interface device. 
     
     
       8. The method of  claim 1 , further comprising presenting the movement signal to the subject in the form of at least one of visual movement of the biological interface device, visual movement of a cursor on a display of the biological interface device, visual representation of control, auditory representation of control, vibrotactile stimulation, and electrical stimulation. 
     
     
       9. The method of  claim 1 , wherein re-training the decoder function comprises obtaining a weight value associated with the feedback signal and re-training the decoder function in accordance with the weight value. 
     
     
       10. The method of  claim 9 , wherein the weight value is determined responsive to historical feedback signals. 
     
     
       11. The method of  claim 1 , further comprising presenting the predictive signal to the subject in the form of at least one of visual movement of the biological interface device, visual movement of a cursor on a display of the biological interface device, visual representation of control, auditory representation of control, vibrotactile stimulation, and electrical stimulation. 
     
     
       12. The method of  claim 1 , further comprising obtaining a training set of values from the historical input signals and the historical predictive signals. 
     
     
       13. The method of  claim 12 , further comprising updating decoder parameters with a weighted combination of data from a new training set of values and a past training set of values. 
     
     
       14. The method of  claim 13 , wherein a weight of the weighted combination is determined by at least one of uncertainty and performance of the decoder function. 
     
     
       15. The method of  claim 1 , further comprising re-training the decoder function at predetermined intervals comprising at least one of fixed intervals, adaptive intervals determined by the operation of the biological interface device, and periodic intervals determined by a user. 
     
     
       16. The method of  claim 15 , further comprising presenting a target to the subject through a visual display, augmented reality display, or virtual reality display;
 obtaining the input signal from the subject, the input signal associated with a command for operation of the biological interface device in interaction with the target; 
 wherein re-training the decoder function comprises re-training the decoder function responsive to data from the input set of values associated with the interaction with the target. 
 
     
     
       17. The method of  claim 1 , wherein the predictive signal comprises an eye tracking signal and an environmental visual signal. 
     
     
       18. The method of  claim 17 , comprising operating a display device of the biological interface device with the output set of values to make a selection from a menu. 
     
     
       19. The method of  claim 17 , comprising operating a mechanical device of the biological interface device with the output set of values to actuate motion of the mechanical device. 
     
     
       20. The method of  claim 1 , wherein eye-tracking signals are utilized for estimating subject intent to be used for error-correction with the decoder function. 
     
     
       21. The method of  claim 1 , wherein the predictive signal includes an involuntary physiological signal and the input signal includes a voluntary physiological signal. 
     
     
       22. The method of  claim 1 , further comprising updating decoder parameters with a weighted combination of data from a new training set of values and a past training set of values, wherein a weight of the weighted combination is determined by uncertainty of the decoder function, such that greater weights are associated with lesser uncertainty. 
     
     
       23. The method of  claim 22 , further comprising utilizing a Gaussian filter to estimate the uncertainty of the decoder function. 
     
     
       24. A biological interface system comprising:
 a subject signal sensor configured to collect at least one neural signal and at least one physiological signal from a subject, the at least one physiological signal measuring physical movement of the subject; 
 an environmental signal sensor configured to collect an environmental signal from an environment of the subject; 
 a signal processor operably connected to the subject signal sensor and the environmental signal sensor, the signal processor configured to generate an input set of values associated with the at least one neural signal or a first physiological signal and a predictive set of values associated with a second physiological signal or the environmental signal; 
 a feedback signal sensor configured to collect an actual signal from operation of the biological interface system operably connected to the signal processor, the signal processor configured to generate a feedback set of values from the actual signal; 
 a memory device electrically connected to the signal processor and storing data from the input set of values, and the predictive set of values, and the feedback set of values; and 
 a system processor coupled to the memory device and configured to
 perform at least one calculation on the input set of values and on the predictive set of values using a decoder function trained with stored historical data to produce an output set of values, the historical data including a matrix F of features and a matrix K of desired kinematic control signals for providing the output set of values, such that the matrix K is constructed by matrix multiplication of F by a matrix of regression coefficients C which represents weighted contribution of the features, 
 operate a biological interface device with the output set of values, and 
 determine a difference between the feedback set of values and the predictive set of values and re-train the decoder function if the difference between the feedback set of values and the predictive set of values is greater than a predetermined threshold, thereby performing co-adaptation to incorporate the actual signal to iteratively improve the decoder function while the subject is attempting control via the biological interface device. 
 
 
     
     
       25. The system of  claim 24 , wherein the subject signal sensor comprises at least one of a microphone, a motion sensor, a temperature sensor, a light sensor, a camera, a chemical sensor, a galvanic skin response sensor, a heart rate monitor, a blood pressure monitor, an external electrode grid, an intracranial electrode grid, an intraneural electrode grid, and an intramuscular electrode grid. 
     
     
       26. The system of  claim 24 , wherein the environmental signal sensor comprises at least one of a microphone, a motion sensor, a temperature sensor, a light sensor, a camera, a chemical sensor, a global positioning system, a clock, and an orientation sensor. 
     
     
       27. The system of  claim 24 , wherein the biological interface device comprises at least one of a mechanical device and an electronic device. 
     
     
       28. The system of  claim 24 , further comprising a display device configured to display a visual indication of at least one of the neural signal, the at least one physiological signal, the environmental signal, and an actual signal associated with operation of the biological interface device. 
     
     
       29. The system of  claim 24 , further comprising a display device configured to display a target.