Electronic device and method for spatial synchronization of videos

Abstract

An electronic device is provided that determines initial three-dimensional (3D) coordinates of a lighting device. The electronic device controls an emission of light from the lighting device based on control signals. The emitted light includes at least one of a pattern of alternating light pulses or a continuous light pulse. The electronic device controls a plurality of imaging devices to capture a first plurality of images that include information about the emitted light. Based on the determined initial 3D coordinates and the information about the emitted light included in the first plurality of images, the electronic device estimates a plurality of rotation values and a plurality of translation values of each imaging device. Based on the plurality of rotation values and the plurality of translation values, the electronic device applies a simultaneous localization and mapping process for each imaging device, for spatial synchronization of the plurality of imaging devices.

What is claimed is: 
     
       1. An electronic device, comprising:
 circuitry communicatively coupled to a plurality of imaging devices and a lighting device, wherein the circuitry is configured to:
 determine initial three-dimensional (3D) coordinates of the lighting device, wherein the lighting device includes a grid of lights and an edge light; 
 control an emission of light from the lighting device based on at least one control signal, wherein the emitted light includes at least one of a pattern of alternating light pulses or a continuous light pulse; 
 control the plurality of imaging devices to capture a first plurality of images that include information about the emitted light; 
 estimate a plurality of rotation values and a plurality of translation values of each imaging device, based on the determined initial 3D coordinates of the lighting device and the information about the emitted light; 
 determine a first set of images of the first plurality of images that includes information about the pattern of alternating light pulses of the emitted light; 
 determine a center of each light of the grid of lights of the lighting device in a first set of frames of the first set of images, wherein the first set of frames includes an ON pattern of the pattern of alternating light pulses; 
 estimate, based on the initial 3D coordinates of the lighting device and the determined center of each light of the grid of lights in the first set of frames, a first rotation value and a first translation value of the plurality of rotation values and the plurality of translation values for each imaging device of the plurality of imaging devices; and 
 apply a simultaneous localization and mapping (SLAM) process for each imaging device of the plurality of imaging devices, based on the plurality of rotation values and the plurality of translation values, for spatial synchronization of the plurality of imaging devices. 
 
 
     
     
       2. The electronic device according to  claim 1 , wherein the circuitry is further configured to:
 generate a synchronization signal that includes a preamble pulse and a sequence of alternating ON and OFF pulse, wherein the at least one control signal includes the synchronization signal; and 
 control the emission of light from the lighting device based on the generated synchronization signal that includes the pattern of alternating light pulses corresponding to the generated synchronization signal. 
 
     
     
       3. The electronic device according to  claim 2 , wherein
 the circuitry is further configured to generate the synchronization signal based on a set of parameters corresponding to each imaging device of the plurality of imaging devices, and 
 the set of parameters includes at least a frame rate of each imaging device of the plurality of imaging devices. 
 
     
     
       4. The electronic device according to  claim 2 , wherein for the control of the emission of the light from the lighting device based on the generated synchronization signal, the circuitry is further configured to:
 activate the grid of lights of the lighting device to generate an ON pattern of the pattern of alternating light pulses of the emitted light; 
 deactivate the grid of lights of the lighting device to generate an OFF pattern of the pattern of alternating light pulses of the emitted light; and 
 deactivate the edge light of the lighting device. 
 
     
     
       5. The electronic device according to  claim 1 , wherein the circuitry is further configured to control the plurality of imaging devices to capture the first set of images based on a determination that the lighting device is in a field-of-view of a respective imaging device of the plurality of imaging devices. 
     
     
       6. The electronic device according to  claim 1 , wherein the circuitry is further configured to:
 apply a set of post-processing operations on the first set of frames of the first plurality of images; and 
 determine the center of each light of the grid of lights of the lighting device based on the applied set of post-processing operations on the first set of frames. 
 
     
     
       7. The electronic device according to  claim 1 , wherein the circuitry is further configured to:
 apply a neural network model on the first set of frames of the first plurality of images to determine a first frame, wherein the first frame includes the information about the pattern of alternating light pulses; and 
 determine the center of each light of the grid of lights of the lighting device in the determined first frame of the first set of frames. 
 
     
     
       8. The electronic device according to  claim 1 , wherein the circuitry is further configured to:
 control the lighting device to activate the edge light of the lighting device; 
 control a transformation of the lighting device towards each imaging device of the plurality of imaging devices, wherein the transformation includes at least one of a rotation or a translation of the lighting device; 
 control, based on the transformation of the lighting device, the plurality of imaging devices to capture light emitted by the edge light; 
 receive a second plurality of images from the plurality of imaging devices, wherein the received second plurality of images includes information about the light emitted by the edge light; and 
 estimate a slope of the information about the light emitted by the edge light in the second plurality of images. 
 
     
     
       9. The electronic device according to  claim 8 , wherein the circuitry is further configured to:
 determine, based on the estimated slope of the information about the light emitted by the edge light in the second plurality of images, a set of grid lines that pass through the determined center of each light of the grid of lights of the lighting device in the first set of frames; 
 determine, based on an intersection of the determined set of grid lines, at least one projected 2D position of the center of each light in the first set of frames; and 
 estimate, based on the initial 3D coordinates of the lighting device and the determined at least one projected 2D position of the center of each light, the first rotation value and the first translation value of the plurality of rotation values and the plurality of translation values of each imaging device of the plurality of imaging devices. 
 
     
     
       10. The electronic device according to  claim 1 , wherein the circuitry is further configured to estimate, based on a perspective-n-point (PnP) technique, the first rotation value and the first translation value for each imaging device of the plurality of imaging devices. 
     
     
       11. The electronic device according to  claim 9 , wherein the circuitry is further configured to determine, based on a mathematical optimization function, the set of grid lines that pass through the determined center of each light of the grid of lights in the first set of frames. 
     
     
       12. The electronic device according to  claim 9 , wherein the circuitry is further configured to determine, based on a neural network model, the set of grid lines that pass through the determined center of each light of the grid of lights in the first set of frames. 
     
     
       13. The electronic device according to  claim 1 , wherein the circuitry is further configured to control the emission of the light from the lighting device based on a specific control signal, wherein
 the emitted light includes the continuous light pulse corresponding to the specific control signal, and 
 the at least one control signal includes the specific control signal. 
 
     
     
       14. The electronic device according to  claim 13 , wherein, the circuitry is further configured to:
 for the control of the emission of the light from the lighting device based on the specific control signal:
 activate the grid of lights of the lighting device to emit the light; and 
 deactivate the edge light of the lighting device. 
 
 
     
     
       15. The electronic device according to  claim 1 , wherein the circuitry is further configured to:
 control a transformation of each imaging device of the plurality of imaging devices for a first time period, wherein each imaging device captures a second set of images of the first plurality of images; 
 determine the center of each light of the grid of lights in the second set of images, wherein the second set of images includes information about the emitted light that includes the continuous light pulse; 
 estimate, based on the determined initial 3D coordinates of the lighting device and the determined center of each light of the grid of lights of the lighting device in the second set of images, a second rotation value and a second translation value of the estimated plurality of rotation values and the plurality of translation values for each imaging device of the plurality of imaging devices; and 
 apply the SLAM process for each imaging device, based on the estimated second rotation value and the second translation value, for the spatial synchronization of the plurality of imaging devices. 
 
     
     
       16. The electronic device according to  claim 15 , wherein the circuitry is further configured to control, based on a determination that the lighting device is in a field-of-view of a respective imaging device of the plurality of imaging devices, each imaging device of the plurality of imaging devices to capture the second set of images. 
     
     
       17. A method, comprising:
 in an electronic device communicatively coupled to a plurality of imaging devices and a lighting device:
 determining initial three-dimensional (3D) coordinates of the lighting device, wherein the lighting device includes a grid of lights and an edge light; 
 controlling an emission of light from the lighting device based on at least one control signal, wherein the emitted light includes at least one of a pattern of alternating light pulses or a continuous light pulse; 
 controlling the plurality of imaging devices to capture a first plurality of images that include information about the emitted light; 
 estimating a plurality of rotation values and a plurality of translation values of each imaging device, based on the determined 3D coordinates of the lighting device and the information about the emitted light of images; 
 determining a first set of images of the first plurality of images that includes information about the pattern of alternating light pulses of the emitted light; 
 determining a center of each light of the grid of lights of the lighting device in a first set of frames of the first set of images, wherein the first set of frames includes an ON pattern of the pattern of alternating light pulses; 
 estimating, based on the initial 3D coordinates of the lighting device and the determined center of each light of the grid of lights in the first set of frames, a first rotation value and a first translation value of the plurality of rotation values and the plurality of translation values for each imaging device of the plurality of imaging devices; and 
 applying a simultaneous localization and mapping (SLAM) process for each imaging device of the plurality of imaging devices, based on the plurality of rotation values and the plurality of translation values, for spatial synchronization of the plurality of imaging devices. 
 
 
     
     
       18. The method according to  claim 17 , further comprising:
 generating a synchronization signal that includes a preamble pulse and a sequence of alternating ON and OFF pulse, wherein the at least one control signal includes the synchronization signal; and 
 controlling the emission of light from the lighting device based on the generated synchronization signal that includes the pattern of alternating light pulses corresponding to the generated synchronization signal. 
 
     
     
       19. A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by an electronic device, causes the electronic device to execute operations, the operations comprising:
 determining initial three-dimensional (3D) coordinates of a lighting device coupled to the electronic device, wherein the lighting device includes a grid of lights and an edge light; 
 controlling an emission of light from the lighting device based on at least one control signal, wherein the emitted light includes at least one of a pattern of alternating light pulses or a continuous light pulse; 
 controlling a plurality of imaging devices, coupled to the electronic device, to capture a first plurality of images that include information about the emitted light; 
 estimating a plurality of rotation values and a plurality of translation values of each imaging device, based on the determined 3D coordinates of the lighting device and the information about the emitted light; 
 determining a first set of images of the first plurality of images that includes information about the pattern of alternating light pulses of the emitted light; 
 determining a center of each light of the grid of lights of the lighting device in a first set of frames of the first set of images, wherein the first set of frames includes an ON pattern of the pattern of alternating light pulses; 
 estimating, based on the initial 3D coordinates of the lighting device and the determined center of each light of the grid of lights in the first set of frames, a first rotation value and a first translation value of the plurality of rotation values and the plurality of translation values for each imaging device of the plurality of imaging devices; and 
 applying a simultaneous localization and mapping (SLAM) process for each imaging device of the plurality of imaging devices, based on the plurality of rotation values and the plurality of translation values, for spatial synchronization of the plurality of imaging devices.