Optimized visualization streaming for vehicle environment visualization

Abstract

In various examples, sensor data may be captured by sensors of an ego-object, such as a vehicle traveling in a physical environment, and a representation of the sensor data may be streamed from the ego-object to a remote location to facilitate various remote experiences, such as streaming to a remote viewer (e.g., a friend or relative), streaming to a remote or fleet operator, streaming to a mobile app configured to self-park or summon an ego-object, rendering a 3D augmented reality (AR) or virtual reality (VR) representation of the physical environment, and/or others. In some embodiments, the stream includes one or more command channels used to control data collection, rendering, stream content, or even vehicle maneuvers, such as during an emergency, self-park, or summon scenario.

What is claimed is: 
     
         1 . A method comprising:
 issuing a remote command to an ego-object in a physical environment from a remote device in a remote location outside the physical environment;   receiving, from the ego-object, using the remote device, a stream of data representative of the physical environment, the stream of data being generated using sensor data that was captured by one or more sensors of the ego-object based at least on the remote command; and   causing the remote device to present a visualization of the data stream representative of the physical environment.   
     
     
         2 . The method of  claim 1 , wherein the stream of data representative of the physical environment comprises, for at least one time slice of one or more time slices: a projection or a stitched image of the physical environment, a three-dimensional (3D) model of the physical environment, or a surround view visualization generated using the 3D model. 
     
     
         3 . The method of  claim 1 , wherein the ego-object is a vehicle, the remote device is operated by a remote fleet operator, and the data stream representative of the physical environment received by the remote device comprises a first video feed from outside the vehicle and a second video feed from inside a cabin of the vehicle. 
     
     
         4 . The method of  claim 1 , wherein the remote command from the remote device designates a viewport, and the data stream representative of the physical environment comprise, for at least one time slice of one or more time slices: a surround view visualization of the physical environment generated by the ego-object using the viewport designated by the remote command. 
     
     
         5 . The method of  claim 1 , wherein the data stream representative of the physical environment comprises directional audio steered by the ego-object toward a direction: i) associated with a salient event detected in the physical environment by the ego-object, or designated by the remote command. 
     
     
         6 . The method of  claim 1 , wherein the data stream representative of the physical environment received by the remote device comprises a view, rendered by the ego-object, of a projection of LiDAR or RADAR data captured by the ego-object. 
     
     
         7 . The method of  claim 1 , wherein the data stream representative of the physical environment comprises a visualization of the environment rendered through a viewport directed to view a salient event detected in the physical environment by the ego-object. 
     
     
         8 . The method of  claim 1 , wherein the visualization of the data stream representative of the physical environment comprises a three-dimensional augmented reality or virtual reality representation of the physical environment rendered on the remote device. 
     
     
         9 . The method of  claim 1 , wherein the remote command is a command summoning the ego-object or instructing the ego-object to self-park, and the data stream representative of the physical environment comprises a video feed generated by the ego-object while self-maneuvering in response to the remote command. 
     
     
         10 . The method of  claim 1 , wherein the method is performed by at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing digital twin operations;   a system for performing light transport simulation;   a system for performing collaborative content creation for 3D assets;   a system for performing deep learning operations;   a system for performing real-time streaming;   a system for presenting one or more of augmented reality content, virtual reality content, or mixed reality content;   a system implemented using an edge device;   a system implemented using a robot;   a system for performing conversational AI operations;   a system for generating synthetic data;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.   
     
     
         11 . A processor comprising:
 one or more circuits to:
 issue, to an ego-object in a physical environment, from a remote device in a remote location outside the physical environment, a remote command defining a viewport; 
 receive, from the ego-object, by the remote device, a stream of surround view visualizations rendered by the ego-object through the viewport defined by the remote command; and 
 cause a display controlled by the remote device to present the stream of surround view visualizations of the physical environment. 
   
     
     
         12 . The processor of  claim 11 , wherein the stream further includes directional audio steered by the ego-object toward a direction: i) associated with a salient event detected in the physical environment by the ego-object, or designated by the remote command. 
     
     
         13 . The processor of  claim 11 , the one or more circuits further to:
 detect, by the remote device, a salient event in the physical environment using sensor data received in the stream from the ego-object during a prior time slice;   orient the viewport to face the salient event detected in the physical environment by the remote device; and   issue the remote command directing the ego-object to render the surround view visualizations through the viewport.   
     
     
         14 . The processor of  claim 11 , wherein the display controlled by the remote device is an augmented or virtual reality (AR/VR) headset, the one or more circuits further to cause the AR/VR headset to present a three-dimensional representation of the surround view visualizations of the physical environment. 
     
     
         15 . The processor of  claim 11 , wherein the processor is comprised in at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing digital twin operations;   a system for performing light transport simulation;   a system for performing collaborative content creation for 3D assets;   a system for performing deep learning operations;   a system for performing real-time streaming;   a system for presenting one or more of augmented reality content, virtual reality content, or mixed reality content;   a system implemented using an edge device;   a system implemented using a robot;   a system for performing conversational AI operations;   a system for generating synthetic data;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center; or   a system implemented at least partially using cloud computing resources.   
     
     
         16 . A system comprising:
 one or more processing units to:
 issue a remote command to a vehicle in a physical environment from a remote device in a remote location outside the physical environment; 
 receive, from the vehicle, by the remote device, a stream of data representing the physical environment and generated by the vehicle based at least on the remote command; and 
 cause a display controlled by the remote device to present a visualization of the data representing the physical environment. 
   
     
     
         17 . The system of  claim 16 , wherein the data representing the physical environment comprises, for each time slice of one or more time slices: a projection or stitched image of the physical environment, a three-dimensional (3D) model of the physical environment, or a surround view visualization generated using the 3D model. 
     
     
         18 . The system of  claim 16 , wherein the remote device is operated by a remote fleet operator, and the data representing the physical environment comprises a first video feed from outside the vehicle and a second video feed from inside a cabin of the vehicle. 
     
     
         19 . The system of  claim 16 , wherein the remote command is a command summoning the vehicle or instructing the vehicle to self-park, and the data representing the physical environment comprises a video feed generated by the vehicle while self-maneuvering in response to the remote command. 
     
     
         20 . The system of  claim 16 , wherein the system is comprised in at least one of:
 a control system for an autonomous or semi-autonomous machine;   a perception system for an autonomous or semi-autonomous machine;   a system for performing simulation operations;   a system for performing digital twin operations;   a system for performing deep learning operations;   a system for performing real-time streaming;   a system for presenting one or more of augmented reality content, virtual reality content, or mixed reality content;   a system implemented using an edge device;   a system implemented using a robot;   a system incorporating one or more virtual machines (VMs);   a system implemented at least partially in a data center;   a system for performing light transport simulation;   a system for performing collaborative content creation for 3D assets;   a system for generating synthetic data; or   a system implemented at least partially using cloud computing resources.