Calibration of cameras on unmanned aerial vehicles using human joints

Abstract

A system and method for calibration of cameras on Unmanned Aerial Vehicles (UAVs) is provided. The system receives a set of anchor images of a human subject from a set of anchor cameras and a group of images of the human subject from multiple viewpoints in three-dimensional (3D) space from a group of cameras on a group of UAVs. The system determines a first set of two-dimensional (2D) positions of human joints from the set of anchor images and a second set of 2D positions of the human joints from the group of images. The system computes, as 3D key-points, 3D positions of the human joints based on triangulation using the first set of 2D positions and determines a 2D re-projection error between the 3D key-points and the second set of 2D positions. Thereafter, by minimizing the 2D re-projection error, the system calibrates each camera of the group of cameras.

What is claimed is: 
     
       1. A system, comprising:
 circuitry communicatively coupled to a set of anchor camera devices and a group of cameras mounted on or integrated with a group of Unmanned Aerial Vehicles (UAVs), wherein the circuitry is configured to:
 receive, from the set of anchor camera devices, a set of anchor images of a human subject; 
 receive, from the group of cameras, a group of images of the human subject from multiple viewpoints in three-dimensional (3D) space; 
 determine, for the human subject in each anchor image of the received set of anchor images, a first set of two-dimensional (2D) positions of human joints; 
 determine, for the human subject in each image of the received group of images, a second set of 2D positions of the human joints; 
 compute, as 3D key-points, 3D positions of the human joints in the 3D space based on a triangulation using the determined first set of 2D positions of the human joints; 
 determine a 2D re-projection error between the 3D key-points and the determined second set of 2D positions; and 
 calibrate each camera of the group of cameras by minimizing the determined 2D re-projection error. 
 
 
     
     
       2. The system according to  claim 1 , wherein the 3D space is associated with one of: an outdoor space, an indoor space, or a studio-setup for volumetric capture. 
     
     
       3. The system according to  claim 1 , wherein the circuitry is further configured to:
 control the group of UAVs to move the group of UAVs at the multiple viewpoints in the 3D space; and 
 control the group of cameras mounted on or integrated with the group of UAVs to acquire the group of images of the human subject from the multiple viewpoints. 
 
     
     
       4. The system according to  claim 3 , wherein the set of anchor camera devices comprises at least one camera which is mounted on or integrated with a UAV and which is configured to maintain a fixed pose while the group of cameras is controlled to move relative to the fixed pose to acquire the group of images. 
     
     
       5. The system according to  claim 3 , wherein the set of anchor camera devices comprises at least one pre-calibrated camera which is fixed to a position in the 3D space while the group of cameras is controlled to acquire the group of images. 
     
     
       6. The system according to  claim 3 , wherein the set of anchor camera devices comprises at least one pre-calibrated camera movably coupled to a remotely-controlled camera movement assembly. 
     
     
       7. The system according to  claim 1 , wherein the circuitry is further configured to control the set of anchor camera devices in the 3D space to acquire the set of anchor images. 
     
     
       8. The system according to  claim 1 , wherein the circuitry is configured to determine the first set of 2D positions of the human joints by application of a Machine Learning (ML) model on each anchor image of the received set of anchor images, and wherein the ML model, as a human joint detection framework, comprises a neural network trained on a 2D human joint detection task. 
     
     
       9. The system according to  claim 1 , wherein the circuitry is configured to determine the second set of 2D positions of the human joints by application of an ML model on each image of the received group of images, and wherein the ML model, as a human joint detection framework, comprises a neural network trained on a 2D human joint detection task. 
     
     
       10. The system according to  claim 1 , wherein
 the calibration of each cameras of the group of cameras corresponds to estimation of a 3D pose of the corresponding camera, and 
 the 3D pose comprises a 3D position and an orientation of the corresponding camera in the 3D space. 
 
     
     
       11. The system according to  claim 1 , wherein
 at least one UAV of the group of UAVs comprises a position sensor, and 
 the position sensor is one of: Global Navigation Satellite System (GNSS) receiver, an Inertial Measurement Unit (IMU), a sensing camera, an Infra-red marker sensor, or a pattern code scanner. 
 
     
     
       12. The system according to  claim 11 , wherein the circuitry is configured to calibrate each camera of the group of cameras further based on absolute position information acquired from the position sensor of a corresponding UAV. 
     
     
       13. The system according to  claim 1 , wherein the circuitry is configured to calibrate each camera of the group of cameras further based on values of intrinsic calibration parameters for the corresponding camera. 
     
     
       14. A method, comprising:
 receiving, from a set of anchor camera devices, a set of anchor images of a human subject; 
 receiving, from a group of cameras mounted on or integrated with a group of UAVs, a group of images of the human subject from multiple viewpoints in three-dimensional (3D) space; 
 determining, for the human subject in each anchor image of the received set of anchor images, a first set of two-dimensional (2D) positions of human joints; 
 determining, for the human subject in each image of the received group of images, a second set of 2D positions of the human joints; 
 computing, as 3D key-points, 3D positions of the human joints in the 3D space based on a triangulation using the determined first set of 2D positions of the human joints; 
 determining a 2D re-projection error between the 3D key-points and the determined second set of 2D positions; and 
 calibrating each camera of the group of cameras by minimizing the determined 2D re-projection error. 
 
     
     
       15. The method according to  claim 14 , further comprising:
 controlling the group of UAVs to move the group of UAVs at the multiple viewpoints in the 3D space; and 
 controlling the group of cameras mounted on or integrated with the group of UAVs to acquire the group of images of the human subject from the multiple viewpoints. 
 
     
     
       16. The method according to  claim 14 , further comprising:
 determining the first set of 2D positions of the human joints by application of a Machine Learning (ML) model on each anchor image of the received set of anchor images; and 
 determining the second set of 2D positions of the human joints by application of the ML model on each image of the received group of images, and wherein the ML model, as a human joint detection framework, comprises a neural network trained on a 2D human joint detection task. 
 
     
     
       17. The method according to  claim 14 , wherein
 the calibration of each camera corresponds to estimation of a 3D pose of the corresponding UAV, and 
 the 3D pose comprises a 3D position and an orientation of the corresponding UAV in the 3D space. 
 
     
     
       18. The method according to  claim 14 , wherein
 at least one UAV of the group of UAVs comprises a position sensor, and 
 the position sensor is one of: Global Navigation Satellite System (GNSS) receiver, an Inertial Measurement Unit (IMU), a sensing camera, an Infra-red marker sensor, or a pattern code scanner. 
 
     
     
       19. The method according to  claim 18 , further comprising calibrating each UAV of the group of UAVs further based on absolute position information acquired from the position sensor of the corresponding camera-fitted UAV. 
     
     
       20. A non-transitory computer-readable medium having stored thereon computer implemented instructions that, when executed by a computer in a system, causes the system to execute operations, the operations comprising:
 receiving, from a set of anchor camera devices, a set of anchor images of a human subject; 
 receiving, from a group of cameras mounted on or integrated with a group of UAVs, a group of images of the human subject from multiple viewpoints in three-dimensional (3D) space; 
 determining, for the human subject in each anchor image of the received set of anchor images, a first set of two-dimensional (2D) positions of human joints; 
 determining, for the human subject in each image of the received group of images, a second set of 2D positions of the human joints; 
 computing, as 3D key-points, 3D positions of the human joints in the 3D space based on a triangulation using the determined first set of 2D positions of the human joints; 
 determining a 2D re-projection error between the 3D key-points and the determined second set of 2D positions; and 
 calibrating each camera of the group of cameras by minimizing the determined 2D re-projection error.