EMG speech signal detection with feedback

Abstract

Systems and methods are provided for performing EMG signal operations. The system accesses one or more blocks of EMG data that were generated based on a plurality of EMG channels of an EMG communication device based on one or more subthreshold activity (STA) of one or more muscles associated with speech production. The system processes the one or more blocks of the EMG data and computes a metric for each block of the one or more blocks of the EMG data that have been processed. The system determines that the metric representing at least one EMG channel of the plurality of EMG channels transgresses the threshold for detection of the STA and generates audible or visual feedback to indicate that the metric representing the at least one EMG channel of the plurality of EMG channels transgresses the threshold for detection of the STA.

What is claimed is: 
     
         1 . A method comprising:
 accessing one or more blocks of electromyograph (EMG) data, the one or more blocks of the EMG data having been generated by a plurality of EMG channels of an EMG communication device based on one or more subthreshold activity (STA) of one or more muscles associated with speech production;   processing the one or more blocks of the EMG data;   computing a metric for each block of the one or more blocks of the EMG data that have been processed;   determining that the metric representing at least one EMG channel of the plurality of EMG channels transgresses a threshold for detection of the STA;   generating audible or visual feedback to indicate that the metric representing the at least one EMG channel of the plurality of EMG channels transgresses the threshold for detection of the STA, the one or more blocks of the EMG data being generated based on one or more EMG signals received by the EMG communication device comprising a microphone and one or more speakers, the microphone being used to capture the one or more blocks of the EMG data from EMG electrodes responsive to user input, the one or more blocks of the EMG data captured by the EMG electrodes being processed by a machine learning model to generate the audible or visual feedback;   determining that an additional set of blocks of EMG data is associated with a corresponding metric that fails to transgress the threshold for detection of the STA; and   in response to determining that the additional set of blocks of EMG data is associated with a corresponding metric that fails to transgress the threshold for detection of the STA, generating a notification for a user indicating that inner speech failed to be detected and requesting the user to repeat a word corresponding to the inner speech.   
     
     
         2 . The method of  claim 1 , wherein the STA is generated in response to inner speech of the user, the one or more blocks of the EMG data being accessed in response to an activation command comprising the user input, the activation command comprising a spoken command or selection of a button. 
     
     
         3 . The method of  claim 1 , further comprising:
 prompting the user of the EMG communication device to perform actual speech by presenting a first message to verbalize a word, phrase, or sentence;   capturing first EMG signals generated in response to the first message;   processing the first EMG signals to remove noise and apply band filtering;   computing a first metric associated with the processed first EMG signals; and   setting the threshold for detection of the STA based on the first metric wherein the metric.   
     
     
         4 . The method of  claim 3 , further comprising:
 subsequently prompting the user to perform imaginary speech by presenting a second message to imagine speaking without verbalizing a word, phrase, or sentence;   capturing second EMG signals generated in response to the second message;   processing the second EMG signals to remove noise and apply band filtering;   computing a second metric associated with the processed second EMG signals;   comparing the second metric with the threshold for detection of the STA;   determining whether the second metric transgresses the threshold for detection of the STA to detect presence or absence of STA;   acquiring, by the EMG communication device, EMG signals from the one or more muscles of the user using one or more electrodes;   transmitting the acquired EMG signals to a server over a communication network, wherein the server generates the one or more blocks of the EMG data in response to receiving the acquired EMG signals; and   wherein accessing the one or more blocks of EMG data comprises receiving the one or more blocks from the server.   
     
     
         5 . The method of  claim 4 , further comprising:
 changing a color or visual attribute of the first message after setting the threshold for detection of the STA based on the first metric to inform the user that the STA has been detected; and   changing a color or visual attribute of the second message to inform the user about the presence or absence of STA based on whether the second metric transgresses the threshold for detection of the STA, wherein processing the one or more blocks of the EMG data comprises:   performing noise reduction and feature extraction operations on the one or more blocks.   
     
     
         6 . The method of  claim 5 , further comprising:
 reducing noise from the one or more blocks of the EMG data; and   after reducing the noise, removing non-informative frequency bands represented by the one or more blocks of the EMG data.   
     
     
         7 . The method of  claim 6 , wherein reducing the noise comprises:
 selecting a first block of EMG data of the one or more blocks of the EMG data;   obtaining a window of blocks of EMG data corresponding to a first time point that precedes a time point of the one or more blocks of the EMG data; and   combining the first block of EMG data with the window of blocks of the EMG data to generate a multi-block signal.   
     
     
         8 . The method of  claim 7 , further comprising:
 applying a short-time Fourier transform to the multi-block signal to generate a signal spectrogram.   
     
     
         9 . The method of  claim 8 , further comprising:
 generating a time-smoothed version of a spectrogram magnitude of the signal spectrogram by applying to a low pass filter with a controllable time constant to each frequency channel.   
     
     
         10 . The method of  claim 9 , further comprising:
 applying the time-smoothed version of the spectrogram magnitude as a noise threshold to identify and attenuate signal components that fail to transgress the noise threshold.   
     
     
         11 . The method of  claim 10 , further comprising:
 applying an activation function to a normalized difference between the spectrogram magnitude of the signal spectrogram and the time-smoothed version of a spectrogram magnitude to map resulting values between a specified range and generate a time-frequency soft-mask.   
     
     
         12 . The method of  claim 11 , wherein the activation function comprises at least one of an ReLU function or sigmoid function. 
     
     
         13 . The method of  claim 11 , further comprising:
 applying an inverse short-time Fourier transform to obtain a time-domain enhanced multi-block signal; and   selecting a last set of samples from the time-domain enhanced multi-block signal as the one or more blocks of the EMG data with reduced noise.   
     
     
         14 . The method of  claim 13 , further comprising:
 applying a band-pass filter to each block in the time-domain enhanced multi-block signal to generate a band-limited block.   
     
     
         15 . The method of  claim 7 , further comprising:
 updating the window of blocks of EMG data with the first block of EMG data after reducing the noise, wherein updating the window of blocks comprises removing an oldest block from the window of blocks.   
     
     
         16 . The method of  claim 1 , further comprising:
 performing a calibration session to compute the threshold.   
     
     
         17 . The method of  claim 16 , further comprising:
 defining the threshold as a function of a scaling constant and a scalar-valued function that provides an indication of STA;   prompting a user of the EMG communication device to perform actual and imaginary speech; and   in response to prompting the user to perform imaginary speech, calculating the scalar-valued function to set the threshold for STA.   
     
     
         18 . The method of  claim 17 , further comprising:
 adaptively modifying the scaling constant based on a noise level of an environment of the EMG communication device.   
     
     
         19 . A system comprising:
 a storage device of an electromyograph (EMG) communication device; and   at least one processor of the EMG communication device configured to perform operations comprising:   accessing one or more blocks of EMG data, the one or more blocks of the EMG data having been generated by a plurality of EMG channels of an EMG communication device based on one or more subthreshold activity (STA) of one or more muscles associated with speech production;   processing the one or more blocks of the EMG data;   computing a metric for each block of the one or more blocks of the EMG data that have been processed;   determining that the metric representing at least one EMG channel of the plurality of EMG channels transgresses a threshold for detection of the STA;   generating audible or visual feedback to indicate that the metric representing the at least one EMG channel of the plurality of EMG channels transgresses the threshold for detection of the STA, the one or more blocks of the EMG data being generated based on one or more EMG signals received by the EMG communication device comprising a microphone and one or more speakers, the microphone being used to capture the one or more blocks of the EMG data from EMG electrodes responsive to user input, the one or more blocks of the EMG data captured by the EMG electrodes being processed by a machine learning model to generate the audible or visual feedback;   determining that an additional set of blocks of EMG data is associated with a corresponding metric that fails to transgress the threshold for detection of the STA; and   in response to determining that the additional set of blocks of EMG data is associated with a corresponding metric that fails to transgress the threshold for detection of the STA, generating a notification for a user indicating that inner speech failed to be detected and requesting the user to repeat a word corresponding to the inner speech.   
     
     
         20 . A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
 accessing one or more blocks of electromyograph (EMG) data, the one or more blocks of the EMG data having been generated by a plurality of EMG channels of an EMG communication device based on one or more subthreshold activity (STA) of one or more muscles associated with speech production;   processing the one or more blocks of the EMG data;   computing a metric for each block of the one or more blocks of the EMG data that have been processed;   determining that the metric representing at least one EMG channel of the plurality of EMG channels transgresses a threshold for detection of the STA;   generating audible or visual feedback to indicate that the metric representing the at least one EMG channel of the plurality of EMG channels transgresses the threshold for detection of the STA, the one or more blocks of the EMG data being generated based on one or more EMG signals received by the EMG communication device comprising a microphone and one or more speakers, the microphone being used to capture the one or more blocks of the EMG data from EMG electrodes responsive to user input, the one or more blocks of the EMG data captured by the EMG electrodes being processed by a machine learning model to generate the audible or visual feedback;   determining that an additional set of blocks of EMG data is associated with a corresponding metric that fails to transgress the threshold for detection of the STA; and   in response to determining that the additional set of blocks of EMG data is associated with a corresponding metric that fails to transgress the threshold for detection of the STA, generating a notification for a user indicating that inner speech failed to be detected and requesting the user to repeat a word corresponding to the inner speech.