Systems and methods to increase environment awareness associated with remote driving applications

Abstract

Systems and methods to increase environment awareness in remote driving applications may include a vehicle having an imaging device and a teleoperator station in communication with each other via a network. For example, audio data may be received from the vehicle and processed to identify known sounds associated with unseen objects in the environment. In addition, imaging data may be received from the vehicle and processed to identify known but unheard objects in the environment. Based on the identified sounds and/or objects, visualizations of the objects may be generated and presented to a teleoperator, and sounds associated with the objects may be amplified, synthesized, and/or emitted to the teleoperator to increase environment awareness.

What is claimed is: 
     
         1 . A method to increase environment awareness in a remote driving system, comprising:
 receiving, by a processor at a teleoperator station via a communication network, video data from an imaging device associated with a vehicle, the vehicle being positioned within an environment remote from the teleoperator station;   receiving, by the processor, audio data captured by a microphone array associated with a vehicle;   processing, by the processor, the audio data to identify a known sound and a relative location of the known sound with respect to the vehicle;   determining, by the processor, a relevant object associated with the known sound at the relative location;   generating, by the processor, a visualization of the relevant object;   causing presentation, via a presentation device at the teleoperator station, of the visualization of the relevant object overlaid onto the video data; and   causing emission, via the presentation device, of the known sound associated with the relevant object.   
     
     
         2 . The method of  claim 1 , wherein the known sound is identified using at least one of an audio pattern matching algorithm, a machine learning model, or a neural network. 
     
     
         3 . The method of  claim 1 , further comprising:
 instructing, by the processor, beamforming of the microphone array to capture the audio data at the relative location with respect to the vehicle.   
     
     
         4 . The method of  claim 1 , wherein the visualization of the relevant object comprises at least one of a visual indicator, a symbol, a color, or a bounding box. 
     
     
         5 . The method of  claim 1 , further comprising:
 amplifying, by the processor, the known sound associated with the relevant object.   
     
     
         6 . A method, comprising:
 receiving, by a processor associated with a teleoperator station via a network, imaging data from an imaging device associated with a vehicle, the vehicle being positioned within an environment remote from the teleoperator station;   receiving, by the processor, audio data captured by an audio sensor associated with a vehicle;   processing, by the processor, the audio data to identify a sound;   determining, by the processor, an object associated with the sound;   generating, by the processor, a visualization of the object;   causing presentation, via a presentation device associated with the teleoperator station, of the visualization of the object within the imaging data; and   causing emission, via the presentation device, of the sound associated with the object.   
     
     
         7 . The method of  claim 6 , wherein the audio sensor comprises an audio sensor array. 
     
     
         8 . The method of  claim 7 , further comprising:
 processing, by the processor, the audio data from the audio sensor array to determine a location of the sound relative to the vehicle;   wherein the object associated with the sound is determined based at least in part on the location of the sound.   
     
     
         9 . The method of  claim 7 , further comprising:
 instructing, by the processor, beamforming of the audio sensor array to capture the audio data at a location relative to the vehicle.   
     
     
         10 . The method of  claim 6 , wherein the sound is identified using at least one of an audio pattern matching algorithm, a machine learning model, or a neural network. 
     
     
         11 . The method of  claim 6 , wherein the visualization of the object comprises at least one of a visual indicator, a symbol, a color, or a bounding box. 
     
     
         12 . The method of  claim 6 , wherein the visualization of the object is presented within the imaging data at a position based on a location of the sound relative to the vehicle. 
     
     
         13 . The method of  claim 6 , further comprising:
 prior to causing emission of the sound associated with the object:
 amplifying, by the processor, the sound associated with the object. 
   
     
     
         14 . The method of  claim 6 , further comprising:
 prior to causing emission of the sound associated with the object:
 generating, by the processor, a synthetic sound associated with the object; 
 wherein causing emission of the sound comprises causing emission of the synthetic sound. 
   
     
     
         15 . The method of  claim 6 , further comprising:
 processing, by the processor, the imaging data to identify the object; and   processing, by the processor, the imaging data to determine a location of the object.   
     
     
         16 . A remote driving system, comprising:
 a vehicle within an environment, the vehicle comprising an imaging device and an audio sensor; and   a teleoperator station that is remote from the vehicle, the teleoperator station in communication with the vehicle via a communication network, the teleoperator station comprising a control interface, a presentation device, and a processor;   wherein the processor is configured to at least:
 receive imaging data from the imaging device associated with the vehicle; 
 receive audio data captured by the audio sensor associated with the vehicle; 
 process the audio data to identify a sound; 
 determine an object associated with the sound; 
 generate a visualization of the object; 
 cause presentation, via the presentation device, of the visualization of the object within the imaging data; and 
 cause emission, via the presentation device, of the sound associated with the object. 
   
     
     
         17 . The remote driving system of  claim 16 , wherein the audio sensor comprises an audio sensor array;
 wherein the processor is further configured to at least:
 process the audio data from the audio sensor array to determine a location of the sound relative to the vehicle; 
 wherein the object associated with the sound is determined based at least in part on the location of the sound. 
   
     
     
         18 . The remote driving system of  claim 16 , wherein the audio sensor comprises an audio sensor array;
 wherein the processor is further configured to at least:
 instruct beamforming of the audio sensor array to capture the audio data at a location relative to the vehicle. 
   
     
     
         19 . The remote driving system of  claim 16 , wherein the processor is further configured to at least:
 determine a position within the imaging data of the visualization of the object based on a location of the sound relative the vehicle;   wherein the visualization of the object is presented at the position within the imaging data.   
     
     
         20 . The remote driving system of  claim 16 , wherein the processor is further configured to at least:
 prior to causing emission of the sound associated with the object:
 amplify or synthesize the sound associated with the object.