Neuromuscular electrical stimulation controlled by computer vision

Abstract

An assistance method for assisting a person in grasping or otherwise manipulating an object includes receiving video of a hand of the person and of an object. An intent to grasp the object is identified based on proximity of the hand to the object in the video or as measured by a proximity sensor, or using gaze tracking, or based on measured neural activity of the person. The object and the hand in the video are analyzed to determine an object grasping action for grasping or otherwise manipulating the object. An actuator is controlled to cause the hand to perform the determined hand action for grasping or otherwise manipulating the object.

The invention claimed is: 
     
         1 . An rehabilitation assistance system comprising:
 a video camera arranged to acquire video of a hand of a person and of an object;   a functional electrical stimulation (FES) device comprising a sleeve configured to be worn on the hand and/or on an arm to which the hand is attached and having surface electrodes arranged on an inner surface of the sleeve to electrically contact the hand and/or arm when the sleeve is worn on the hand and/or the arm of the person; and   an electronic processor programmed to:
 identify an intent to manipulate the object; 
 determine a hand action for manipulating the object based on analysis of the video; and 
 control the FES device to cause the hand to perform the determined hand action for manipulating the object. 
   
     
     
         2 . The rehabilitation assistance system of  claim 1  wherein the identification of the intent to manipulate the object comprises identification of an intent to grasp and/or lift and/or move the object and the determination of the hand action comprises determining an object grasping action for grasping the object. 
     
     
         3 . The rehabilitation assistance system of  claim 2  wherein the electronic processor is programmed to determine the object grasping action for grasping the object by operations including:
 identifying the object in the video; and 
 determining the object grasping action based on the identification of the object. 
 
     
     
         4 . The rehabilitation assistance system of  claim 2  wherein the electronic processor is programmed to determine the object grasping action for grasping the object by operations including:
 segmenting the object in at least one frame of the video; and 
 determining the object grasping action based on a shape of the segmented object. 
 
     
     
         5 . The rehabilitation assistance system of  claim 1  wherein the electronic processor is further programmed to:
 segment the object in at least one frame of the video; 
 segment the hand in the at least one frame of the video; and 
 determine a hand-object spatial relationship based on the segmented object and the segmented hand; 
 wherein the determination of the hand action for manipulating the object is based on the determined hand-object spatial relationship. 
 
     
     
         6 . The rehabilitation assistance system of  claim 1  wherein the identification of the intent to manipulate the object comprises an intent to move the object and the determination of the hand action comprises determining an object grasping action for grasping the object and the electronic processor is further programmed to:
 control the FES device to cause the hand to release the object grasping action in response to analysis of the video indicating a completion of the movement of the object. 
 
     
     
         7 . The rehabilitation assistance system of  claim 1  further comprising:
 a force sensor or proximity sensor disposed on a bottom of the object; 
 wherein the identification of the intent to manipulate the object comprises an intent to move the object and the determination of the hand action comprises determining an object grasping action for grasping the object and the electronic processor is further programmed to control the FES device to cause the hand to release the object grasping action in response to receiving a signal indicating a completion of the movement of the object from the force sensor or proximity sensor disposed on the bottom of the object. 
 
     
     
         8 . The rehabilitation assistance system of  claim 1  wherein the electronic processor is further programmed to:
 detect an action trigger based on proximity of the hand to the object in the video or as measured by a proximity sensor attached to one or both of the hand and the object; 
 wherein the control of the FES device is performed in response to the proximity being closer than a threshold. 
 
     
     
         9 . The rehabilitation assistance system of  claim 1  wherein the determined hand action includes a hand orientation action and an object interaction action, and the electronic processor is further programmed to:
 detect a first action trigger based on a first proximity of the hand to the object in the video or as measured by a proximity sensor attached to one or both of the hand and the object; and 
 detect a second action trigger based on a second proximity of the hand to the object in the video or as measured by the proximity sensor attached to one or both of the hand and the object; 
 wherein the second proximity is closer than the first proximity; and 
 wherein the control of the FES device includes controlling the FES device to cause the hand to perform the hand orientation action in response to the first action trigger and controlling the FES device to cause the hand to perform the object interaction action in response to the second action trigger. 
 
     
     
         10 . The rehabilitation assistance system of  claim 1  wherein the electronic processor is programmed to identify the intent to manipulate the object based on at least one of proximity of the hand to the object in the video, proximity of the hand to the object measured by a proximity sensor, a measured gaze of the person focusing on the object, and a measured neural activity of the person. 
     
     
         11 . The rehabilitation assistance system of  claim 1  wherein:
 the electronic processor is programmed to identify the intent to manipulate the object including at least one of intent to grip the object and/or intent to release a grip on the object based on at least a measured neural activity of the person comprising brain activity measured by an electroencephalogram (EEG) or a brain-computer interface (BCI) and/or surface electromyography (EMG) measured by the FES device, the measured neural activity not being used to determine the hand action for manipulating the object. 
 
     
     
         12 . The rehabilitation assistance system of  claim 1  wherein:
 the electronic processor is programmed to identify the intent to manipulate the object including at least one of intent to grip the object and/or intent to release a grip on the object based on surface electromyography (EMG) measured by the FES device. 
 
     
     
         13 . The rehabilitation assistance system of  claim 1  further comprising smart glasses, wherein the video camera is mounted on or in the smart glasses. 
     
     
         14 . An assistance method comprising:
 acquiring video of a hand of a person and of an object;   identifying an intent to manipulate the object based on at least one of proximity of the hand to the object in the video, proximity of the hand to the object measured by a proximity sensor, a measured gaze of the person focusing on the object, and a measured neural activity of the person;   determining, from a set of hand actions stored in a lookup table, a hand action for manipulating the object based on analysis of the object and the hand in the video performed by an electronic processor; and   using the electronic processor, controlling a functional electrical stimulation (FES) device comprising a sleeve worn at least on the hand and/or an arm of the person and having surface electrodes arranged on an inner surface of the sleeve to electrically contact the hand and/or arm to cause the hand to perform the determined hand action for manipulating the object.   
     
     
         15 . The activity assistance method of  claim 14  wherein the identifying of the intent to manipulate the object comprises identifying an intent to grasp and/or lift and/or move the object and the determining of the hand action comprises determining an object grasping action for grasping the object. 
     
     
         16 . The activity assistance method of  claim 15  wherein the determining of the object grasping action for grasping the object includes one of:
 identifying the object in the video and determining the object grasping action based on the identification of the object; or 
 segmenting the object in at least one frame of the video and determining the object grasping action based on a shape of the segmented object; or 
 segmenting the object in at least one frame of the video, segmenting the hand in the at least one frame of the video, and determining a hand-object spatial relationship based on the segmented object and the segmented hand, wherein the object grasping action for grasping the object is determined based on the hand-object spatial relationship. 
 
     
     
         17 . The activity assistance method of  claim 15  further comprising:
 determining a proximity of the hand to the object in the video; 
 wherein the controlling of the FES device is performed in response to the proximity of the hand to the object in the video being closer than a threshold. 
 
     
     
         18 . A non-transitory storage medium storing instructions that are readable and executable by an electronic processor to perform an assistance method including:
 receiving video of a hand of a person and of an object;   identifying an intent to grasp the object based on at least one of proximity of the hand to the object in the video, proximity of the hand to the object measured by a proximity sensor, a measured gaze of the person focusing on the object, and a measured neural activity of the person;   analyzing the object and the hand in the video to determine an object grasping action for grasping the object from a set of object grasping actions stored in a lookup table; and   controlling a functional electrical stimulation (FES) device comprising a sleeve worn at least on the hand and/or an arm of the person and having surface electrodes arranged on an inner surface of the sleeve to electrically contact the hand and/or arm to cause the hand to perform the determined hand action for grasping the object.   
     
     
         19 . The non-transitory storage medium of  claim 18  wherein the determined object grasping action for grasping the object includes a hand orientation action and a hand gripping action, and the controlling of the FES device includes:
 controlling the FES device to perform the hand orientation action in response to proximity of the hand to the object in the video being less than a first threshold; and 
 controlling the FES device to perform the hand gripping action in response to proximity of the hand to the object in the video being less than a second threshold that is less than the first threshold. 
 
     
     
         20 . The non-transitory storage medium of  claim 18  wherein:
 the intent to grasp is identified based on at least a measured neural activity of the person; and 
 the object grasping action for grasping the object is not determined based on the measured neural activity of the person.