Systems and methods for augmented stereoscopic display

Abstract

A method includes, with aid of one or more processors individually or collectively, analyzing stereoscopic video data of an environment to determine environmental information, generating augmented stereoscopic video data of the environment by fusing the stereoscopic video data and the environmental information, and controlling an unmanned aerial vehicle (UAV) to avoid an obstacle on a motion path of the UAV according to the augmented stereoscopic video data of the environment.

What is claimed is: 
     
         1 . A method comprising, with aid of one or more processors individually or collectively:
 analyzing stereoscopic video data of an environment to determine environmental information;   generating augmented stereoscopic video data of the environment by fusing the stereoscopic video data and the environmental information; and   controlling an unmanned aerial vehicle (UAV) to avoid an obstacle on a motion path of the UAV according to the augmented stereoscopic video data of the environment.   
     
     
         2 . The method of  claim 1 , wherein the environmental information includes information of the obstacle in the environment, the information of the obstacle including at least one of a distance of the UAV from the obstacle or an orientation of the UAV relative to the obstacle. 
     
     
         3 . The method of  claim 1 , wherein the environmental information includes an environmental map, the environmental map including a topological map or a metric map. 
     
     
         4 . The method of  claim 1 , wherein the environmental information includes at least one of:
 relative positions between the UAV and one or more objects in the environment; or   relative positions between two or more objects in the environment.   
     
     
         5 . The method of  claim 1 , wherein the environmental information includes at least one of:
 a distance of the UAV from a target in the environment, or   an orientation of the UAV relative to the target.   
     
     
         6 . The method of  claim 5 , wherein the UAV is configured to follow or track the target. 
     
     
         7 . The method of  claim 1 , wherein analyzing the stereoscopic video data includes at least one of stereo matching of image frames or depth map calculation. 
     
     
         8 . The method of  claim 7 , wherein the stereo matching includes matching of feature points extracted from the image frames based on motion characteristics of the feature points. 
     
     
         9 . The method of  claim 8 , further comprising:
 detecting a position of the obstacle in the environment based on a depth image obtained by matching a left-eye image and a right-eye image.   
     
     
         10 . The method of  claim 7 , further comprising:
 determining the obstacle by filtering and applying a threshold to a depth map in the depth map calculation.   
     
     
         11 . The method of  claim 10 , wherein the threshold is applied to classify the obstacle in the environment having at least one of a predetermined size or a number of pixels in the depth map. 
     
     
         12 . The method of  claim 1 , wherein generating the augmented stereoscopic video data of the environment by fusing the stereoscopic video data and the environmental information includes:
 generating an augmented reality (AR) layer that includes the environmental information;   correcting a distortion of images corresponding to the stereoscopic video data based on the environmental information that is obtained through a sensing unit onboard the UAV and includes geometric information about at least one object within a field of view of an imaging device of the UAV; and   aligning the AR layer to the corrected images to generate, on a terminal remote from the UAV, the augmented stereoscopic video data of the environment.   
     
     
         13 . The method of  claim 12 , wherein:
 wherein the AR layer further includes display information of a reference line that extends across a first person view (FPV) of the environment displayed on a terminal and is relative to a horizontal line of the FPV;   the method further comprising:
 in response to a change in at least a pitch or a roll of the UAV, obtaining updated environmental information to generate an updated AR layer, the updated AR layer including updated display information of the reference line indicative of a change in an orientation of the reference line relative to the horizontal line. 
   
     
     
         14 . The method of  claim 1 , wherein the augmented stereoscopic video data of the environment includes a graphical element associated with the environmental information. 
     
     
         15 . The method of  claim 14 , wherein the graphical element is configured to dynamically change as a position or the orientation of the UAV relative to the obstacle in the environment changes. 
     
     
         16 . The method of  claim 14 , wherein the graphical element is overlay on a first person view (FPV) of the environment displayed on a terminal. 
     
     
         17 . The method of  claim 16 , wherein the terminal includes at least one of a head-mounted display (HMD), a pair of virtual reality (VR) or augmented reality (AR) enabled glasses, or a mobile device mounted at a foldable headgear and including a graphical display configured to display the FPV. 
     
     
         18 . The method of  claim 16 , wherein the terminal allows a user to control and navigate the UAV from the FPV. 
     
     
         19 . The method of  claim 1 , wherein fusing the stereoscopic video data and the environmental information includes incorporating the environmental information into the stereoscopic video data. 
     
     
         20 . An unmanned aerial vehicle (UAV) comprising:
 a central body;   a binocular video camera configured to generate stereoscopic video data of an environment; and   one or more processors, individually or collectively configured to:
 receive the stereoscopic video data of the environment; 
 analyze the stereoscopic video data to determine environmental information; 
 generate augmented stereoscopic video data of the environment by fusing the stereoscopic video data and the environmental information; and 
 control the UAV to avoid an obstacle on a motion path of the UAV according to the augmented stereoscopic video data of the environment.