Methods and systems for human-in-the-loop vehicular command and control using immersive synthetic vision

Abstract

The present invention provides a synthetic vision system to support the piloting of a vehicle. The system may comprise: an immersive display screen and/or head-mounted virtual reality display, one or more real-time sources of information from onboard the vehicle (e.g., aircraft), one or more data sources that are not onboard the vehicle, either stored in memory or accessed in real-time via a communications channel, and a computer with a graphics processing unit to combine multiple sources of data and render them in real-time to be shown to the pilot via the synthetic vision system display.

1 . A system for providing synthetic vision to a human operator, the system comprising:
 a display device disposed at a control station remote from a movable object capable of translational and rotational movement;   one or more processors configured to perform operations including:
 receiving real-time data from one or more data sources and accessing data stored in a local or cloud storage device to construct a virtual view; and 
 rendering the virtual view to the human operator via the display device for controlling an operation of the movable object under Visual Flight Rules (VFR) condition or Instrument Flight Rules (IFR) condition, 
 wherein the virtual view comprises a first-person view (FPV) or a third-person view (TPV) and wherein either the FPV or the TPV comprises at least a rendering of a natural object serving as a reference point to the human operator. 
   
     
     
         2 . The system of  claim 1 , wherein the movable object comprises a fly-by-wire control system for controlling an actuator of the movable object in response to a command received from the control station. 
     
     
         3 . The system of  claim 2 , wherein the movable object is a helicopter. 
     
     
         4 . The system of  claim 1 , wherein the virtual view is displayed based on measurements of a movement of the human operator's head and/or eyes. 
     
     
         5 . The system of  claim 1 , wherein the real-time data comprise video stream captured by an imaging device onboard the movable object and wherein the natural object is not visible in the video stream. 
     
     
         6 . The system of  claim 1 , wherein the operations further include determining data to be displayed within the virtual view based on the VFR condition or the IFT condition. 
     
     
         7 . The system of  claim 1 , wherein the TPV is configurable by changing a virtual TPV camera location. 
     
     
         8 . The system of  claim 1 , wherein the operations further include activating a transparency mode in the TPV when the movable object is approaching a destination. 
     
     
         9 . The system of  claim 1 , wherein the virtual view comprises a rendering of a dynamic obstacle. 
     
     
         10 . The system of  claim 9 , wherein the dynamic obstacle is tracked by processing sensor data collected from movable object. 
     
     
         11 . The system of  claim 10 , wherein a location of the dynamic obstacle is tracked by applying a feed-forward model to the sensor data. 
     
     
         12 . The system of  claim 11 , wherein an identity of the dynamic obstacle is determined by applying a machine learning algorithm trained model to the sensor data. 
     
     
         13 . The system of  claim 12 , wherein the rendering of the dynamic obstacle is based at least in part on a certainty of the identity and/or the location. 
     
     
         14 . A method for providing synthetic vision to a human operator, the method comprising:
 providing a display device at a control station remote from a movable object capable of translational and rotational movement;   receiving real-time data from one or more data sources and accessing data stored in a local or cloud storage device to construct a virtual view; and   rendering the virtual view to the human operator via the display device for controlling an operation of the movable object under Visual Flight Rules (VFR) condition or Instrument Flight Rules (IFR) condition,   wherein the virtual view comprises a first-person view (FPV) or a third-person view (TPV) and wherein either the FPV or the TPV comprises at least a rendering of a natural object serving as a reference point to the human operator.   
     
     
         15 . The method of  claim 14 , wherein the movable object comprises a fly-by-wire control system for controlling an actuator of the movable object in response to a command received from the control station. 
     
     
         16 . The method of  claim 15 , wherein the movable object is a helicopter. 
     
     
         17 . The method of  claim 14 , wherein the virtual view is displayed based on measurements of a movement of the human operator's head and/or eyes. 
     
     
         18 . The method of  claim 14 , wherein the real-time data comprise video stream captured by an imaging device onboard the movable object and wherein the natural object is not visible in the video stream. 
     
     
         19 . The method of  claim 14 , further comprising determining data to be displayed within the virtual view based on the VFR condition or the IFT condition. 
     
     
         20 . The method of  claim 14 , wherein the TPV is configurable by changing a virtual TPV camera location.