Multisensory speech detection

Abstract

A computer-implemented method of multisensory speech detection is disclosed. The method comprises determining an orientation of a mobile device and determining an operating mode of the mobile device based on the orientation of the mobile device. The method further includes identifying speech detection parameters that specify when speech detection begins or ends based on the determined operating mode and detecting speech from a user of the mobile device based on the speech detection parameters.

1 . A computer-implemented method comprising:
 detecting, using one or more sensors of a mobile computing device, motion of the mobile computing device from a starting position to an ending position in three-dimensional space;   determining, by the mobile computing device and using stored data that characterizes one or more predetermined patterns, whether the detected motion matches one or more patterns that indicate a user of the mobile computing device has performed a voice input gesture, wherein the voice input gesture comprises the mobile computing device being moved near the user's face in preparation for receiving voice input;   in response to determining that the detected motion matches the one or more patterns, activating a microphone of the mobile computing device and initiating a speech detection operation;   monitoring, using the microphone and the speech detection operation, for voice input from the user for at least a threshold period of time;   detecting the voice input during the threshold period of time; and   causing the voice input to be converted to text.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein determining whether the detected motion matches the one or more patterns comprises:
 determining whether acceleration of the mobile computing device, detected while the mobile computing device is moving between the starting position and the ending position, is greater than one or more threshold values that are associated with the one or more patterns.   
     
     
         3 . The computer-implemented method of  claim 1 , wherein determining whether the detected motion matches the one or more patterns comprises:
 determining whether an angle of the mobile computing device, detected with regard to one or more axes and when the mobile computing device is at or near the ending position, is within one or more predefined ranges of angles with respect to vertical that are associated with the one or more patterns.   
     
     
         4 . The computer-implemented method of  claim 1 , wherein determining whether the detected motion matches the one or more patterns comprises:
 determining whether a distance between the mobile computing device and an object that is located in front of a proximity sensor of the mobile computing device, detected when the mobile computing device is at or near the ending position, is within one or more predefined ranges of distance that are associated with the one or more patterns.   
     
     
         5 . The computer-implemented method of  claim 1 , wherein determining whether the detected motion matches the one or more patterns comprises:
 determining whether:
 i) acceleration of the mobile computing device, detected while the mobile computing device is moving between the starting position and the ending position, is greater than one or more threshold values that are associated with the one or more patterns, 
 ii) an angle of the mobile computing device, detected with regard to one or more axes and when the mobile computing device is at or near the ending position, is within one or more predefined ranges of angles that are associated with the one or more patterns, and 
 iii) a distance between the mobile computing device and an object that is located in front of a proximity sensor of the mobile computing device, detected when the mobile computing device is at or near the ending position, is within one or more predefined ranges of distance that are associated with the one or more patterns; 
   wherein the user is determined to have performed the voice input gesture when i) the acceleration is greater than the one or more threshold values, ii) the angle is within the one or more predefined ranges of angles, and iii) the distance is within the one or more predefined ranges of distance.   
     
     
         6 . The computer-implemented method of  claim 1 , further comprising:
 causing a search to be conducted using at least a portion of the text as a search query; and   outputting results for the search query on the mobile computing device.   
     
     
         7 . The computer-implemented method of  claim 6 , wherein causing the search to be conducted comprises:
 causing to be searched, using the search query, one or more data sources that are local to mobile computing device; and   causing to be retrieved at least a portion of the results for the search query from the one or more data sources.   
     
     
         8 . The computer-implemented method of  claim 6 , wherein causing the search to be conducted comprises:
 providing the search query to a computer system that is remote from the mobile computing device; and   receiving at least a portion of the results for the search query from the computer system.   
     
     
         9 . The computer-implemented method of  claim 1 , wherein causing the voice input to be converted to the text occurs by the mobile computing device causing a computer system that is remote from the mobile computing device to convert the voice input to text. 
     
     
         10 . The computer-implemented method of  claim 1 , wherein the voice input is converted to the text by the mobile computing device. 
     
     
         11 . The computer-implemented method of  claim 1 , wherein the one or more sensors comprise a proximity sensor. 
     
     
         12 . The computer-implemented method of  claim 1 , wherein the one or more sensors comprise an accelerometer. 
     
     
         13 . A mobile computing device comprising:
 one or more sensors to detect motion of the mobile computing device from a starting position to an ending position in three-dimensional space;   a gesture classifier to determine whether the detected motion matches one or more patterns that indicate a user of the mobile computing device has performed a voice input gesture, wherein the voice input gesture comprises the mobile computing device being moved near the user's face in preparation for receiving voice input;   a microphone that is activated in response to the gesture classifier determining that the voice input gesture has been performed;   a speech detection module to monitor audio signals received by the microphone for voice input from the user for at least a threshold period of time; and   a speech-to-text conversion module to cause the voice input detected during the threshold period of time to be converted to text.   
     
     
         14 . The mobile computing device of  claim 13 , wherein:
 the one or more sensors include a proximity sensor; and   the gesture classifier is further configured to determine whether a distance between the mobile computing device and an object that is located in front of the proximity sensor, detected when the mobile computing device is at or near the ending position, is within one or more predefined ranges of distance that are associated with the one or more patterns.   
     
     
         15 . The mobile computing device of  claim 13 , wherein:
 the one or more sensors include an accelerometer; and   the gesture classifier is further configured to determine whether acceleration of the mobile computing device, as detected by the accelerometer while the mobile computing device is moving between the starting position and the ending position, is greater than one or more threshold values that are associated with the one or more patterns.   
     
     
         16 . The mobile computing device of  claim 13 , wherein:
 the one or more sensors include an orientation sensor; and   the gesture classifier is further configured to determine whether the angle of the mobile computing device, detected by the orientation sensor with regard to one or more axes and when the mobile computing device is at or near the ending position, is within one or more predefined ranges of angles that are associated with the one or more patterns.   
     
     
         17 . The mobile computing device of  claim 13 , further comprising
 a processor to cause a search to be conducted using at least a portion of the text as a search query; and   one or more output devices to output results for the search query.   
     
     
         18 . The mobile computing device of  claim 17 , wherein the one or more output devices include a display to visually output the results, or one or more speakers to audibly output the results. 
     
     
         19 . The mobile computing device of  claim 13 , wherein the speech-to-text conversion module is arranged to cause the voice input to be converted to text by submitting an audio file of the voice input to a remote server system for speech-to-text conversion and receiving text corresponding to the audio file from the server system. 
     
     
         20 . A computer program product tangibly embodied on a computer readable storage medium storing instructions that, when executed by or more processors of a computing system, cause the computing system to perform operations comprising:
 detecting, using one or more sensors, motion of a computing device from a starting position to an ending position in three-dimensional space;   determining, using stored data that characterizes one or more predetermined patterns, whether the detected motion matches one or more patterns that indicate a user of the computing device has performed a voice input gesture, wherein the voice input gesture comprises the computing device being moved near the user's face in preparation for receiving voice input;   in response to determining that the detected motion matches the one or more patterns, activating a microphone of the computing device and initiating a speech detection operation;   monitoring, using the microphone and the speech detection operation, for voice input from the user for at least a threshold period of time;   detecting the voice input during the threshold period of time; and   causing the voice input to be converted to text.