Method to align an immersive video and an immersive sound field

Abstract

A system comprising a video source, one or more audio sources and a computing device. The video source may be configured to generate a plurality of video streams that capture a view of an environment. The one or more audio sources may be configured to capture audio data of the environment. The computing device may comprise one or more processors configured to (i) perform a stitching operation on the plurality of video streams to generate a video signal representative of an immersive field of view of the environment, (ii) generate a sound field based on the audio data, (iii) identify an orientation for the sound field with respect to the video signal, and (iv) determine a rotation of the sound field based on the orientation. The rotation of the sound field aligns the sound field to the video signal.

1 . A method for sound field alignment for an immersive video signal, comprising the steps of:
 (A) receiving said immersive video signal captured by a video capture device;   (B) receiving one or more audio streams configured to generate a sound field captured by an audio capture device;   (C) identifying an orientation for said captured sound field with respect to said captured immersive video signal; and   (D) determining a rotation of said sound field based on said orientation when said video capture device and said audio capture device do not share an orientation and axis, wherein (a) said rotation of said sound field aligns said sound field to said immersive video signal, and (b) said sound field and said immersive video signal are aligned prior to being transmitted to a playback device.   
     
     
         2 . The method according to  claim 1 , wherein identifying said orientation comprises (i) identifying a first axis for said sound field with respect to said immersive video signal and (ii) identifying a rotation of said sound field around said first axis. 
     
     
         3 . The method according to  claim 1 , wherein identifying said orientation comprises (i) identifying a first axis for said sound field with respect to said immersive video signal and (ii) identifying a second axis for said sound field with respect to said immersive video signal. 
     
     
         4 . The method according to  claim 3 , wherein (i) said sound field is configured to rotate around said first axis and (ii) said second axis is constrained to lie on a pre-determined curve displayed on a representation of said immersive video signal. 
     
     
         5 . The method according to  claim 1 , wherein (a) the origin of axes of said immersive video signal and the origin of axes of said audio streams are (i) aligned and (ii) fixed and (b) said orientation of said axes of said audio streams is determined based on parameters identifying a direction of said audio capture device during a capture of said immersive video signal. 
     
     
         6 . The method according to  claim 1 , wherein said orientation for said sound field is identified based on a plurality of commands received from a user while said user observes a representation of said immersive video signal. 
     
     
         7 . The method according to  claim 6 , wherein said representation of said immersive video signal further comprises a visual representation of a coordinate system corresponding to said sound field. 
     
     
         8 . The method according to  claim 6 , wherein (i) said user identifies a reference object on said representation of said immersive video and (ii) coordinates of said reference object are converted to a longitude value. 
     
     
         9 . The method according to  claim 8 , wherein a graphic is generated on said representation of said immersive video to identify said orientation for said sound field. 
     
     
         10 . The method according to  claim 6 , wherein (i) said representation of said immersive video signal is presented to said user on a motion-tracking display, (ii) said sound field is presented to said user with a binaural audio output device and (iii) said user inputs said commands to identify said orientation for said sound field by bringing an audio source to a frontal position on said representation of said immersive video signal. 
     
     
         11 . The method according to  claim 1 , wherein said rotation is applied to said sound field. 
     
     
         12 . The method according to  claim 1 , wherein parameters are generated to represent said rotation and said parameters are sent to playback device as metadata. 
     
     
         13 . The method according to  claim 1 , wherein an adjustment to said rotation is performed based on an interpolation of a plurality of keyframes. 
     
     
         14 . The method according to  claim 1 , wherein corrective rotations are applied to said sound field based on corresponding corrective rotations to said immersive video signal implemented to perform image stabilization. 
     
     
         15 . The method according to  claim 1 , wherein identifying said orientation and determining said rotation is based on inertial measurements from (i) a first sensor configured to indicate an absolute orientation of said video capture device and (ii) a second sensor configured to indicate an absolute orientation of said audio capture device. 
     
     
         16 . The method according to  claim 1 , wherein said immersive video signal and said sound field are combined in a single stream. 
     
     
         17 . A system comprising:
 a video source configured to generate an immersive video signal;   one or more audio sources configured to generate a sound field; and   a computing device comprising one or more processors configured to (i) identify an orientation for said sound field with respect to said immersive video signal and (ii) determine a rotation of said sound field based on said orientation when said video source and said audio sources do not share an orientation and axis, wherein (a) said rotation of said sound field aligns said sound field to said immersive video signal, and (b) said sound field and said immersive video signal are aligned prior to being transmitted to a playback device.   
     
     
         18 . A system comprising:
 a video source configured to generate a plurality of video streams that capture a view of an environment;   one or more audio sources configured to capture audio data of said environment; and   a computing device comprising one or more processors configured to (i) perform a stitching operation on said plurality of video streams to generate a video signal representative of an immersive field of view of said environment, (ii) generate a sound field based on said audio data, (iii) identify an orientation for said sound field with respect to said video signal when said video source and said audio sources do not share an orientation and axis, and (iv) determine a rotation of said sound field based on said orientation, wherein (a) said rotation of said sound field aligns said sound field to said video signal, and (b) said sound field and said video signal are aligned prior to being transmitted to a playback device.   
     
     
         19 . The method according to  claim 1 , wherein said video capture device comprises a spherical video camera configured to capture an immersive field of view. 
     
     
         20 . The method according to  claim 1 , wherein said audio capture device comprises an ambisonic microphone.