Speechreading using facial feature parameters from a non-direct frontal view of the speaker

Abstract

A system for performing recognition having a telephone transmitter, a camera, a data channel and recognition processing logic, in which the camera is directly mounted to and positioned with respect to the telephone housing to obtain video information from a non-direct frontal view of the speaker corresponding to at least one facial feature for use in speechreading. The facial features that may be obtained include the position of the tongue, separation of the teeth and the rounding protrusion of the lips. Using this data, recognition processing logic performs speechreading recognition of the video information.

I claim: 
     
       1. A system for performing recognition comprising: a telephone transmitter contained in a movable telephone transmitter housing;   a camera directly mounted to and positioned with respect to the telephone transmitter to obtain video information, corresponding to at least one facial feature for speechreading, from a non-direct frontal view of a speaker;   a data channel coupled to the camera to transfer the video information from the camera; and   a recognition processing logic coupled to the data channel to perform speechreading recognition of the video information.   
     
     
       2. The system defined in claim 1 wherein the telephone transmitter housing comprises a telephone handset. 
     
     
       3. The system defined in claim 1 wherein the telephone transmitter housing comprises a telephone headset. 
     
     
       4. The system defined in claim 1 wherein the camera comprises a digital camera. 
     
     
       5. The system defined in claim 1 wherein the video information comprises position of a tongue of a user and the rounding protrusion of the lips. 
     
     
       6. The system defined in claim 5 wherein the video information further comprises the position of the jaw. 
     
     
       7. The system defined in claim 6 wherein the position of the jaw is based on separation of teeth. 
     
     
       8. The system defined in claim 1 further comprising a light source mounted on the telephone device to illuminate a user's mouth. 
     
     
       9. The system defined in claim 1 further comprising an infrared (IR) source mounted to the telephone device to illuminate a user's mouth. 
     
     
       10. The system defined in claim 9 wherein the camera comprises an IR-sensitive camera. 
     
     
       11. The system defined in claim 9 wherein the camera comprises an optical camera. 
     
     
       12. An apparatus for obtaining data for use by recognition processing logic, said apparatus comprising: a telephone transmitter;   a camera coupled to the telephone transmitter and positioned with respect to the telephone transmitter to obtain video information, corresponding to at least one facial feature for speechreading, from a non-direct frontal view of a speaker, wherein the video information comprises position of a tongue of a user and the rounding protrusion of the lips;   a data channel coupled to the camera to transfer the video information from the camera to the recognition processing logic to enable speechreading recognition of the video information.   
     
     
       13. The apparatus defined in claim 12 wherein the video information further comprises the position of the jaw. 
     
     
       14. The apparatus defined in claim 13 wherein the position of the jaw is based on separation of teeth. 
     
     
       15. The apparatus defined in claim 12 wherein the camera comprises a digital camera. 
     
     
       16. The apparatus defined in claim 12 further comprising a light source mounted to the telephone device to illuminate a user's mouth. 
     
     
       17. The apparatus defined in claim 12 further comprising an infrared (IR) source mounted to the telephone device to illuminate a user's mouth. 
     
     
       18. The apparatus defined in claim 17 wherein the camera comprises an IR-sensitive camera. 
     
     
       19. The apparatus defined in claim 17 wherein the camera comprises an IR optical-sensitive camera. 
     
     
       20. A method for performing recognition comprising the steps of: receiving audio information from a speaker using a telephone transmitter;   receiving video information, corresponding to at least one facial feature of the speaker, from a non-direct frontal view of the speaker, using a camera coupled to the telephone transmitter;   transferring the audio and video information by a data channel to recognition logic for speech and pattern recognition.   
     
     
       21. The system defined in claim 1 wherein the camera images a portion of the speaker's mouth from a location which is at an angle with respect to a direct frontal view of the speaker that is dependent on the speaker's facial features and the speaker's positioning of the telephone transmitter. 
     
     
       22. The apparatus defined in claim 12 wherein the camera images a portion of the speaker's mouth from a location which is at an angle with respect to a direct frontal view of the speaker that is dependent on the speaker's facial features and the speaker's positioning of the telephone transmitter.