System and method for capturing omni-stereo videos using multi-sensors

Abstract

A system and method for capturing Omni-Stereo videos using multi-sensor is disclosed. The system includes left cameras, right cameras and a viewing circle. The method of capturing omni stereo videos using multi-sensor approach includes steps of: capturing images of a scene using left cameras, capturing images of a scene using right cameras, processing each image from the left camera and right camera using a computation method, and obtaining a final omni stereo frame through the computation method.

1 . A system for capturing an omni-stereo image, comprising:
 a multi sensor unit that comprises (a) at least three left eye cameras comprising a first left eye camera; a second left eye camera and a third left eye camera and (b) at least three right eye cameras comprising a first right eye camera; a second right eye camera and a third right eye camera, wherein said at least three left eye cameras and said at least three right eye cameras are arranged along a viewing circle or viewing ellipse and are angled tangentially with respect to the viewing circle or the viewing ellipse;   a memory that stores a set of machine-readable instructions; and   a processor that executes the set of machine-readable instructions to
 calibrate the at least three left eye cameras and the at least three right eye cameras; 
 compute a left eye panorama based on images captured by the at least three left eye cameras and a right eye panorama based on images captured by the at least three right eye cameras; and 
 combine the left eye panorama and the right eye panorama to obtain the omni-stereo image. 
   
     
     
         2 . The system of  claim 1 , wherein the left eye panorama is computed without requiring stitching or compositing of the images captured by the at least three left eye cameras, or wherein the right eye panorama is computed without requiring stitching or compositing of images captured by the at least three right eye cameras. 
     
     
         3 . The system of  claim 1 , wherein a second end of a first right view of the first right eye camera concurs with a first end of a second right view of the second right eye camera and a second end of the second right view of the second right eye camera concurs with a first end of a third right view of the third right eye camera and a second end of the third right view of the third right eye camera concurs with a first end of the first right view of the first right eye camera, wherein the first, second and third right eye cameras are placed tangentially to the viewing circle in a first direction to capture left tangential rays to eliminate missing area or overlap between the first right view, the second right view and the third right view and, wherein a second end of a first left view of the first left eye camera concurs with a first end of a third left view of the third left eye camera and a second end of the third left view of the third left eye camera concurs with a first end of a second left view of the second left eye camera and a second end of the second left view of the second left eye camera concurs with a first end of the first left view of the first left eye camera, wherein the first, second and third left eye cameras are placed tangentially to the viewing circle in a second direction to capture left tangential rays to eliminate missing parts or overlap between the first left view, the second left view and the third left view. 
     
     
         4 . The system of  claim 1 , wherein the at least three left eye cameras are directed clockwise around the viewing circle or the viewing ellipse, and wherein the at least three right eye cameras a directed counter-clockwise around the viewing circle or the viewing ellipse. 
     
     
         5 . The system of  claim 1 , wherein the multi sensor unit comprises three left eye cameras and three right eye cameras, wherein each camera includes at least 120 degrees of usable view that is aligned with usable view of other cameras in same direction arranged along the viewing circle or the viewing ellipse. 
     
     
         6 . The system of  claim 1 , wherein the multi sensor unit comprises six left eye cameras and six right eye cameras, wherein each camera includes at least 60 degrees of usable view that is aligned with usable view of other cameras in same direction arranged along the viewing circle or the viewing ellipse. 
     
     
         7 . The system of  claim 1 , wherein a field of view of the omni-stereo image is at least 360 degrees horizontal (H)×180 degrees vertical (V). 
     
     
         8 . The system of  claim 1 , wherein the images captured by the at least left eye cameras and the at least three right eye cameras of a scene are processed by the computation method, wherein the computation method adds each image of the scene and divides the summed images with 2pi/n to obtain an omni-directional image. 
     
     
         9 . The system of  claim 1 , wherein the multi-sensor unit is configured to be on a movable vehicle to obtain an omni stereo image or video. 
     
     
         10 . (canceled) 
     
     
         11 . The system of  claim 1  wherein an audio is recorded using microphone along omni stereo image or video. 
     
     
         12 . The system of  claim 1 , wherein each of the cameras employs a wide angle lens or a fish eye lens. 
     
     
         13 . A method of forming an omni-stereo image, comprising:
 obtaining images from at least three left eye cameras and at least three right eye cameras arranged along a viewing circle or a viewing ellipse and angled tangentially with respect to the viewing circle or the viewing ellipse;   calibrating the at least three left eye cameras and the at least three right eye cameras;   computing, with aid of one or more processors, a left eye panorama based on images captured by the at least three left eye cameras and a right eye panorama based on images captured by the at least three right eye cameras; and   combining, with aid of the one or more processors, the left eye panorama and the right eye panorama to obtain the omni-stereo image.   
     
     
         14 . The method of  claim 13 , wherein calibrating the at least three left eye cameras and the at least three right eye cameras comprises categorizing pairs of the left eye cameras and the right eye cameras into one of a plurality of categories, and aligning the pairs of the left eye cameras and the right eye cameras by category. 
     
     
         15 . The method of  claim 14 , wherein calibrating the at least three left eye cameras and the at least three right eye cameras comprises converting wide angle images or fisheye images captured by the at least three left eye cameras and the at least three right eye cameras to equirectangular images. 
     
     
         16 . The method of  claim 13 , wherein the left eye panorama is computed without requiring stitching or compositing of the images captured by the at least three left eye cameras, or wherein the right eye panorama is computed without requiring stitching or compositing of images captured by the at least three right eye cameras. 
     
     
         17 . The method of  claim 13 , further comprising calculating a depth image based on the left eye panorama and the right eye panorama. 
     
     
         18 . A method of calibrating cameras used to collect images to form an omni-stereo image, comprising:
 determining intrinsic and extrinsic camera parameters for each of a plurality of left eye cameras and a plurality of right eye cameras arranged along a viewing circle or a viewing ellipse and angled tangentially with respect to the viewing circle or the viewing ellipse;   categorizing, with aid of one or more processors left-right pairs of the plurality of left eye cameras and the plurality of right eye cameras into at least a first category, a second category or a third category;   aligning the left-right pairs of cameras that fall into the first category;   aligning the left-right pairs of cameras that fall into the second category; and   aligning the left-right pairs of cameras that fall into the third category by using extrinsic parameters of the left-right pairs that fall into the first category, and of the left-right pairs that fall into the second category.   
     
     
         19 . The method of  claim 18 , wherein the first category is a straight pair category wherein a left camera and a right camera point in the same direction. 
     
     
         20 . The method of  claim 19 , the second category is a perpendicular pair category wherein a left camera and a right camera point in perpendicular directions with respect to one another, and capture overlapping portions of a scene. 
     
     
         21 . The method of  claim 20 , wherein the third category is a consecutive pair category wherein a left camera and a right camera point in perpendicular directions with respect to one another, and do not capture overlapping portions of the scene.