Exploiting visual information for enhancing audio signals via source separation and beamforming

Abstract

A system for exploiting visual information for enhancing audio signals via source separation and beamforming is disclosed. The system may obtain visual content associated with an environment of a user, and may extract, from the visual content, metadata associated with the environment. The system may determine a location of the user based on the extracted metadata. Additionally, the system may load, based on the location, an audio profile corresponding to the location of the user. The system may also load a user profile of the user that includes audio data associated with the user. Furthermore, the system may cancel, based on the audio profile and user profile, noise from the environment of the user. Moreover, the system may include adjusting, based on the audio profile and user profile, an audio signal generated by the user so as to enhance the audio signal during a communications session of the user.

What is claimed is: 
     
         1 . A method, comprising:
 determining, by a user equipment comprising a processor, metadata of audio captured in an environment in which a communication session is being performed using the user equipment, wherein the metadata comprises information indicating a user identity of a user that is part of the communication session and an environment type of the environment; and   cancelling, by the user equipment, noise in the environment in additional audio, other than the audio, captured during the communication session based on the metadata and an audio profile associated with the environment type.   
     
     
         2 . The method of  claim 1 , wherein the cancelling is further based on a user profile associated with the user identity. 
     
     
         3 . The method of  claim 2 , wherein the user profile identifies an accent associated with the user identity. 
     
     
         4 . The method of  claim 2 , wherein the user profile identifies a tone of a voice associated with the user identity. 
     
     
         5 . The method of  claim 2 , wherein the user profile identifies a pitch of a voice associated with the user identity. 
     
     
         6 . The method of  claim 2 , further comprising adjusting, by the user equipment, an audio signal associated with the user identity in the additional audio based on the user profile. 
     
     
         7 . The method of  claim 1 , wherein the cancelling comprises transmitting a cancellation signal in a direction of a location of a source of the noise. 
     
     
         8 . A user equipment, comprising:
 a processor; and   a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
 obtaining metadata of media captured in an environment in which an audio conversation is being performed using the user equipment, wherein the metadata identifies a user that is part of the audio conversation and an environment type of the environment; and 
 suppressing noise in the environment in additional media captured during the audio conversation based on the metadata and an audio profile associated with the environment type. 
   
     
     
         9 . The user equipment of  claim 8 , wherein the suppressing is further based on a user profile associated with the user. 
     
     
         10 . The user equipment of  claim 9 , wherein the user profile identifies a physical characteristic of the user. 
     
     
         11 . The user equipment of  claim 9 , wherein the user profile identifies a voice characteristic of the user. 
     
     
         12 . The user equipment of  claim 9 , wherein the user profile identifies an age of the user. 
     
     
         13 . The user equipment of  claim 9 , wherein the operations further comprise: enhancing an audio signal associated with the user in the additional media based on the user profile. 
     
     
         14 . The user equipment of  claim 8 , wherein the cancelling comprises separating a first audio signal associated with the noise in the additional media from a second audio associated with the user in the additional media. 
     
     
         15 . A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor of a user equipment, facilitate performance of operations, comprising:
 acquiring metadata of audio captured in an environment in which speech recognition session is being performed using the user equipment, wherein the metadata indicates a user identity that is part of the speech recognition session and an environment type of the environment; and   suppressing noise in the environment in additional audio captured during the speech recognition session based on the metadata and an audio profile associated with the environment type.   
     
     
         16 . The non-transitory machine-readable medium of  claim 15 , wherein the suppressing is further based on a user profile associated with the user identity. 
     
     
         17 . The non-transitory machine-readable medium of  claim 15 , wherein the metadata further indicates respective locations of objects in the environment. 
     
     
         18 . The non-transitory machine-readable medium of  claim 15 , wherein the metadata further indicates respective types of devices being used in the environment. 
     
     
         19 . The non-transitory machine-readable medium of  claim 15 , wherein the metadata further indicates a location of a user, identified by the user identity, in the environment relative to a source of the noise. 
     
     
         20 . The non-transitory machine-readable medium of  claim 15 , wherein the audio profile comprises a noise suppression process trained for the environment type.