Imputation of 3D data using generative adversarial networks

Abstract

A generative adversarial network (GAN) is manufactured by a process including obtaining a three-dimensional (3D) point cloud, extracting a region from the 3D point cloud, the region corresponding to a gap, analyzing the extracted region to generate a loss, backpropagating the loss, and updating weights of the GAN. A computer-implemented method for training a GAN includes obtaining a 3D point cloud, extracting a region from the 3D point cloud, the region corresponding to a gap, analyzing the extracted region to generate a loss, backpropagating the loss, and updating weights of the GAN. A server includes a processor and a memory storing instructions that, when executed by the processor, cause the server to obtain a 3D point cloud, extract a region from the 3D point cloud, the region corresponding to a gap, analyze the extracted region to generate a loss, backpropagate the loss, and update weights of the GAN.

What is claimed: 
     
       1. A non-transitory computer readable storage medium having stored thereon computer instructions that, when executed by one or more processors, cause the one or more processors to:
 obtain one or more training three-dimensional point clouds; 
 extract one or more three-dimensional regions from each training three-dimensional point cloud, wherein extracting the one or more three-dimensional regions from each training three-dimensional point cloud includes creating one or more gaps in each three-dimensional point cloud corresponding to each of the one or more extracted three-dimensional regions, 
 train the generative adversarial network by: 
 analyzing the extracted three-dimensional regions and each three-dimensional point cloud including the respective one or more gaps, wherein the analyzing includes generating a loss value, and 
 updating one or more weights of the generative adversarial network by backpropagating the loss value throughout the generative adversarial network; and 
 store the updated weights of the generative adversarial network on the non-transitory computer readable storage medium as parameters for initializing the generative adversarial network. 
 
     
     
       2. The non-transitory computer readable storage medium of  claim 1 , having stored thereon further instructions to:
 obtain a three-dimensional point cloud having one or more gaps; 
 initialize the generative adversarial network using the stored weights; and 
 impute one or both of (i) RGB data, and (ii) elevation data into the gaps of the three-dimensional point cloud by analyzing the three-dimensional point cloud using the initialized generative adversarial network. 
 
     
     
       3. The non-transitory computer readable storage medium of  claim 2 , having stored thereon further instructions to:
 store the three-dimensional point cloud including the imputed data on the computer readable storage medium. 
 
     
     
       4. The non-transitory computer readable storage medium of  claim 2 , having stored thereon further instructions to:
 generate the three-dimensional point cloud using a structure-from-motion technique. 
 
     
     
       5. The non-transitory computer readable storage medium of  claim 1 , wherein the gaps include one or both of (i) an implicit gap, and (ii) an explicit gap. 
     
     
       6. The non-transitory computer readable storage medium of  claim 1 , having stored thereon further instructions to:
 backpropagate discriminator loss to a discriminator artificial neural network. 
 
     
     
       7. The non-transitory computer readable storage medium of  claim 1 , having stored thereon further instructions to:
 backpropagate discriminator loss to a discriminator artificial neural network and a generator artificial neural network. 
 
     
     
       8. A computer-implemented method for training a generative adversarial network, comprising:
 obtaining one or more training three-dimensional point clouds; 
 extracting one or more three-dimensional regions from each training three-dimensional point cloud, wherein extracting the one or more three-dimensional regions from each training three-dimensional point cloud includes creating one or more gaps in each three-dimensional point cloud corresponding to each of the one or more extracted three-dimensional regions, 
 training the generative adversarial network by:
 analyzing the extracted three-dimensional regions and each three-dimensional point cloud including the respective one or more gaps, wherein the analyzing includes generating a loss value, and 
 updating one or more weights of the generative adversarial network by backpropagating the loss value throughout the generative adversarial network; and 
 
 storing the updated weights of the generative adversarial network on a non-transitory computer readable storage medium as parameters for initializing the generative adversarial network. 
 
     
     
       9. The computer-implemented method of  claim 8 , further comprising:
 obtaining a three-dimensional point cloud having one or more gaps; 
 initializing the generative adversarial network using the stored weights; and 
 imputing one or both of (i) RGB data, and (ii) elevation data into the gaps of the three-dimensional point cloud by analyzing the three-dimensional point cloud using the initialized generative adversarial network. 
 
     
     
       10. The computer-implemented method of  claim 9 , further comprising:
 storing the three-dimensional point cloud including the imputed data on the computer readable storage medium. 
 
     
     
       11. The computer-implemented method of  claim 9 , wherein obtaining the three-dimensional point cloud having one or more gaps includes generating the three-dimensional point cloud using a structure-from-motion technique. 
     
     
       12. The computer-implemented method of  claim 8 , wherein the gaps include one or both of (i) an implicit gap, and (ii) an explicit gap. 
     
     
       13. The computer-implemented method of  claim 8 , wherein updating the one or more weights of the generative adversarial network by backpropagating the loss value throughout the generative adversarial network includes backpropagating discriminator loss to a discriminator artificial neural network. 
     
     
       14. The computer-implemented method of  claim 8 , wherein updating the one or more weights of the generative adversarial network by backpropagating the loss value throughout the generative adversarial network includes backpropagating discriminator loss to a discriminator artificial neural network and a generator artificial neural network. 
     
     
       15. A server comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the server to
 obtain one or more training three-dimensional point clouds; 
 extract one or more three-dimensional regions from each training three-dimensional point cloud, wherein extracting the one or more three-dimensional regions from each training three-dimensional point cloud includes creating one or more gaps in each three-dimensional point cloud corresponding to each of the one or more extracted three-dimensional regions, 
 train the generative adversarial network by:
 analyzing the extracted three-dimensional regions and each three-dimensional point cloud including the respective one or more gaps, wherein the analyzing includes generating a loss value, and 
 updating one or more weights of the generative adversarial network by backpropagating the loss value throughout the generative adversarial network; and 
 
 store the updated weights of the generative adversarial network on a non-transitory computer readable storage medium as parameters for initializing the generative adversarial network. 
 
     
     
       16. The server of  claim 15 , the memory storing further instructions that, when executed by the one or more processors, cause the server to
 obtain a three-dimensional point cloud having one or more gaps; 
 initialize the generative adversarial network using the stored weights; and 
 impute one or both of (i) RGB data, and (ii) elevation data into the gaps of the three-dimensional point cloud by analyzing the three-dimensional point cloud using the initialized generative adversarial network. 
 
     
     
       17. The server of  claim 15 , wherein the gaps include one or both of (i) an implicit gap, and (ii) an explicit gap. 
     
     
       18. The server of  claim 15 , the memory storing further instructions that, when executed by the one or more processors, cause the server to backpropagate discriminator loss to a discriminator artificial neural network. 
     
     
       19. The server of  claim 15 , the memory storing further instructions that, when executed by the one or more processors, cause the server to
 backpropagate discriminator loss to a discriminator artificial neural network and a generator artificial neural network. 
 
     
     
       20. The server of  claim 15 , the memory storing further instructions that, when executed by the one or more processors, cause the server to
 display the three-dimensional point cloud including the imputed data in the display device of a user.