System and method for creating a navigable, three-dimensional virtual reality environment having ultra-wide field of view

Abstract

The present invention relates to a system and method for capturing video of a real-world scene over a field of view that may exceed the field of view of a user, manipulating the captured video, and then stereoscopically displaying the manipulated image to the user in a head mounted display to create a virtual environment having length, width, and depth in the image. By capturing and manipulating video for a field of view that exceeds the field of view of the user, the system and method can quickly respond to movement by the user to update the display allowing the user to look and pan around, i.e., navigate, inside the three-dimensional virtual environment.

1 - 13 . (canceled) 
     
     
         14 . A system for creating a navigable, panoramic three-dimensional virtual reality environment with stereoscopic perception of three-dimensional depth to a user, comprising:
 first and second optical elements each optical element having a horizontal field of view of at least 80 degrees, the first optical element providing a first video image stream and the second optical element providing a second video image stream, each stream corresponding to a different stereoscopic viewpoint of a scene;   a storage device for storing the first and second video image streams, each stream comprising image frames;   an image renderer disposed in communication with the storage device to receive the first and second video image streams, the image renderer configured to create an output stream of merged stereoscopic image frames corresponding to a selected direction of view of the user, the merged frames comprising image frames from the first video image stream which are temporally matched to respective image frames from the second video image stream.   
     
     
         15 . The system according to  claim 14 , wherein the first and second optical elements each include at least one optical imaging detector configured to record the first and second video image streams corresponding to the different stereoscopic viewpoints of the scene. 
     
     
         16 . The system according to  claim 14 , wherein the image renderer is configured to split the merged frames of the output stream and transform each of the split frames onto respective first and second models. 
     
     
         17 . The system according to  claim 16 , wherein the center of origin of each model is selected to correspond to a location at which a respective one of the user's eyes is to be positioned. 
     
     
         18 . The system according to  claim 16 , wherein each model comprises a wireframe model. 
     
     
         19 . The system according to  claim 16 , wherein each model comprises a sphere. 
     
     
         20 . The system according to  claim 14 , wherein the renderer comprises a non-transitory computer usable medium having a computer readable program code embodied therein, the computer readable program code adapted to be executed to implement a method for rendering the first and second video image streams to create a panoramic three-dimensional virtual reality environment having stereoscopic perception of three-dimensional depth comprising the steps of creating first and second wireframe models, and transforming each of the first and second video image streams by wrapping the first and second video image streams onto the first and second wireframe models respectively. 
     
     
         21 . The system according to  claim 14 , wherein the field of view of each optical element is at least 100 degrees. 
     
     
         22 . The system according to  claim 14 , wherein the field of view of each optical element is at least 120 degrees. 
     
     
         23 . The system according to  claim 14 , wherein the field of view of each optical element is at least 150 degrees. 
     
     
         24 . The system according to  claim 14 , wherein the field of view of each optical element is at least 170 degrees. 
     
     
         25 . The system according to  claim 14 , wherein the field of view of each optical element is at least 183 degrees. 
     
     
         26 . The system according to  claim 14 , wherein the field of view of each optical element is at least 210 degrees. 
     
     
         27 . The system according to  claim 14 , wherein the first and second optical elements comprise a first pair of optical elements, wherein the system comprises a second pair of optical elements facing in a different direction from the first pair, each optical element of the second pair having a horizontal field of view of at least 80 degrees, and wherein the first and second pairs of optical dements are configured to provide for the recording of a real-world scene of up to a full 360° by 360° field of view in three dimensions. 
     
     
         28 . A method for rendering video image streams to create a navigable, panoramic three-dimensional virtual reality environment having stereoscopic perception of three-dimensional depth comprising:
 providing the system of  claim 15 ;   capturing the first and second video image streams from a respective one of the at least one imaging detector of the first and second optical elements;   merging selected frames of the first and second video image streams in stereoscopic format to create the output stream of merged stereoscopic image frames;   receiving position data indicating a direction of sight of the user;   calculating a first and second region of interest of the merged stereoscopic image pairs based on the position data; and   displaying the first and second regions of interest on a head mounted stereoscopic display, whereby a navigable, panoramic three-dimensional virtual reality environment having stereoscopic perception of three-dimensional depth is created.   
     
     
         29 . A according to  claim 28 , comprising creating first and second wireframe models, and wrapping respective first and second sections of the merged stereoscopic image frames onto the first and second wireframe models respectively. 
     
     
         30 . A method according to  claim 29 , wherein the displaying of the first and second regions of interest comprises rotating the first and second wireframe models in response to the position data. 
     
     
         31 . A method according to  claim 28 , comprising removing distortion from the first and second image frames. 
     
     
         32 . A method according to  claim 28 , wherein the merging of selected frames of the first and second video image streams comprises temporally synchronizing the first and second video image streams.