Three-dimensional object reconstruction

Abstract

This disclosure relates to reconstructing three-dimensional models of objects from two-dimensional images. According to a first aspect, this specification describes a computer implemented method for creating a three-dimensional reconstruction from a two-dimensional image, the method comprising: receiving a two-dimensional image; identifying an object in the image to be reconstructed and identifying a type of said object; spatially anchoring a pre-determined set of object landmarks within the image; extracting a two-dimensional image representation from each object landmark; estimating a respective three-dimensional representation for the respective two-dimensional image representations; and combining the respective three-dimensional representations resulting in a fused three-dimensional representation of the object.

What is claimed is: 
     
       1. A computer-implemented method for creating a three-dimensional (3D) reconstruction from a two-dimensional (2D) image comprising a plurality of object landmarks corresponding to an object, the method comprising:
 extracting a 2D image representation from each object landmark to generate a plurality of 2D images each corresponding to a different object landmark; 
 estimating a respective 3D representation for each of the plurality of 2D images, each respective separate 3D representation representing a corresponding set of possible orientation angles of the different object landmarks, the possible orientation angles referring to all possible positions of an object landmark; 
 applying a weighting to each orientation angle in the set of possible orientation angles based on a kinematic association of the respective object landmark associated with a respective one of the plurality of separate 3D representations; 
 reducing a weight of the set of possible orientation angles corresponding to one or more object landmarks that are missing from the object; and 
 combining the respective 3D representations comprising the plurality of separate 3D representations of each of the different object landmarks, resulting in a fused 3D representation of the object. 
 
     
     
       2. The method of  claim 1 , further comprising:
 applying a function to the 2D image representation to determine an orientation angle of each of a plurality of joints of the object, the function comprising summations in a numerator and denominator, the one or more object landmarks that are missing from the object being excluded from the summations in the numerator and denominator. 
 
     
     
       3. The method of  claim 1 , wherein spatially anchoring, extracting, estimating and combining are performed in an encoder neural network. 
     
     
       4. The method of  claim 3 , wherein the encoder neural network is one of a plurality of encoder networks, the method further comprising choosing the encoder neural network based on an identified object type. 
     
     
       5. The method of  claim 1 , wherein the combining further comprises:
 decoupling respective estimated 3D representations of object landmarks which are kinematically independent, wherein each respective estimated 3D representation comprises a corresponding set of possible orientation angles of the respective object landmark; and 
 applying a weighting to each orientation angle in the set of possible orientation angles based on a kinematic association of the respective object landmark to the respective possible orientation and visibility of the landmark. 
 
     
     
       6. The method of  claim 1 , further comprising applying the fused 3D representation of the object to a part-based 3D shape representation model. 
     
     
       7. The method of  claim 6 , wherein the part-based 3D shape representation model is a kinematic tree. 
     
     
       8. The method of  claim 1 , further comprising:
 computing a maximum of an output of a 3D joint detection module to identify the set of object landmarks; 
 comparing the maximum to a threshold; and 
 in response to determining that the maximum fails to transgress the threshold, determining that the object landmark of the set of object landmarks corresponds to the one or more object landmarks that are missing from the identified object. 
 
     
     
       9. The method of  claim 1 , wherein each of the plurality of 2D images corresponds to a different joint of the plurality of joints on a human body. 
     
     
       10. The method of  claim 1 , further comprising:
 determining that the one or more object landmarks that are missing from the object are not visible in the 2D image. 
 
     
     
       11. The method of  claim 1 , further comprising assigning a lower weight to the one or more object landmarks that are missing from the object in generating the fused 3D representation of the object. 
     
     
       12. The method of  claim 1 , further comprising:
 determining that a type of object corresponds to an animal; and 
 selecting a neural network specific to the animal to perform estimating the respective 3D representation for the respective 2D image representations. 
 
     
     
       13. The method of  claim 1 , wherein each of the plurality of separate 3D representations represents a corresponding set of possible orientation angles of the different object landmarks, the possible orientation angles referring to all possible positions of an object landmark, further comprising:
 applying a weighting to each orientation angle in the set of possible orientation angles based on a kinematic association of the respective object landmark associated with a respective one of the plurality of separate 3D representations; and 
 reducing a weight of the set of possible orientation angles corresponding to the one or more object landmarks that are missing from the object. 
 
     
     
       14. A system comprising:
 a storage device; and 
 at least one processor coupled to the storage device, wherein at least one processor is configured to perform operations for creating a three-dimensional (3D) reconstruction from a two-dimensional (2D) image comprising a plurality of object landmarks corresponding to an object, the operations comprising: 
 extracting a 2D image representation from each object landmark to generate a plurality of 2D images each corresponding to a different object landmark; 
 estimating a respective 3D representation for each of the plurality of 2D images, each respective separate 3D representation representing a corresponding set of possible orientation angles of the different object landmarks, the possible orientation angles refer to all possible positions of an object landmark; 
 applying a weighting to each orientation angle in the set of possible orientation angles based on a kinematic association of the respective object landmark associated with a respective one of the plurality of separate 3D representations; 
 reducing a weight of the set of possible orientation angles corresponding to one or more object landmarks that are missing from the object; and 
 combining the respective 3D representations comprising the plurality of separate 3D representations of each of the different object landmarks, resulting in a fused 3D representation of the object. 
 
     
     
       15. The system of  claim 14 , wherein the operations comprise:
 computing a maximum of an output of a 2D joint detection module to identify the set of object landmarks; 
 comparing the maximum to a threshold; and 
 in response to determining that the maximum fails to transgress the threshold, determining that the object landmark of the set of object landmarks corresponds to the one or more object landmarks that are missing from the identified object. 
 
     
     
       16. The system of  claim 14 , wherein the operations comprise:
 determining that the one or more object landmarks that are missing from the object are not visible in the 2D image. 
 
     
     
       17. The system of  claim 14 , wherein the operations comprise:
 assigning a lower weight to a set of orientation angles associated with the one or more object landmarks that are missing from the object in generating the fused 3D representation of the object. 
 
     
     
       18. The system of  claim 14 , wherein the operations comprise:
 applying a function to the 2D image representation to determine an orientation angle of each of a plurality of joints of the object, the function comprising summations in a numerator and denominator, the one or more object landmarks that are missing from the object being excluded from the summations in the numerator and denominator. 
 
     
     
       19. The system of  claim 14 , wherein spatially anchoring, extracting, estimating and combining are performed in an encoder neural network. 
     
     
       20. A non-transitory computer readable medium that stores a set of instructions that is executable by at least one processor to perform operations for creating a three-dimensional (3D) reconstruction from a two-dimensional (2D) image comprising a plurality of object landmarks corresponding to an object, the operations comprising:
 extracting a 2D image representation from each object landmark to generate a plurality of 2D images each corresponding to a different object landmark; 
 estimating a respective 3D representation for each of the plurality of 2D images, each respective separate 3D representation representing a corresponding set of possible orientation angles of the different object landmarks, the possible orientation angles referring to all possible positions of an object landmark; 
 applying a weighting to each orientation angle in the set of possible orientation angles based on a kinematic association of the respective object landmark associated with a respective one of the plurality of separate 3D representations; 
 reducing a weight of the set of possible orientation angles corresponding to one or more object landmarks that are missing from the object; and 
 combining the respective 3D representations comprising the plurality of separate 3D representations of each of the different object landmarks, resulting in a fused 3D representation of the object.