Methods and systems for predicting pressure maps of 3D objects from 2D photos using deep learning

Abstract

A structured 3D model of a real-world object is generated from a series of 2D photographs of the object, using photogrammetry, a keypoint detection deep learning network (DLN), and retopology. In addition, object parameters of the object are received. A pressure map of the object is then generated by a pressure estimation DLN based on the structured 3D model and the object parameters. The pressure estimation DLN was trained on structured 3D models, object parameters, and pressure maps of a plurality of objects belonging to a given object category. The pressure map of the real-world object can be used in downstream processes, such as custom manufacturing.

What is claimed is: 
     
       1. A computer-implemented method for generating a pressure map of an object, the computer-implemented method executable by a hardware processor, the method comprising:
 receiving a plurality of 2-dimensional (2D) images of the object, wherein the plurality of 2D images capture the object from different angles; 
 receiving one or more input parameters, the input parameters comprising at least one attribute related to the object; 
 constructing a structured 3-dimensional (3D) model of the object from the plurality of 2D images; and 
 generating the pressure map of the object from the structured 3D model and the input parameters using a pressure estimation deep learning network (DLN), wherein the pressure estimation DLN is trained to generate pressure maps of objects from a given structured 3D model and given parameters of a given object, wherein the pressure map comprises a pressure profile comprising a plurality of pressure points, each pressure point indicating a level of pressure where the object is in contact with a surface, wherein the pressure estimation DLN predicts the level of pressure at the plurality of pressure points. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein constructing the structured 3D model of the object from the plurality of 2D images comprises:
 generating a scaled unstructured 3D mesh of the object from the plurality of 2D images using a photogrammetry process and a scale factor, wherein the scaled unstructured 3D mesh is utilized to generate the structured 3D model. 
 
     
     
       3. The computer-implemented method of  claim 2 , wherein constructing the structured 3D model of the object from the plurality of 2D images further comprises:
 generating the structured 3D model from an annotated scaled unstructured 3D mesh by morphing an annotated structured base 3D mesh to match the annotated scaled unstructured 3D mesh. 
 
     
     
       4. The computer-implemented method of  claim 3 , wherein constructing the structured 3D model of the object from the plurality of 2D images further comprises:
 utilizing a 3D keypoint DLN to generate the annotated scaled unstructured 3D mesh of the object from the scaled unstructured 3D mesh of the object, wherein the annotated scaled unstructured 3D mesh is utilized to generate the structured 3D model. 
 
     
     
       5. The computer-implemented method of  claim 4 , wherein the 3D keypoint DLN is based on a PointNet. 
     
     
       6. The computer-implemented method of  claim 3 , wherein constructing the structured 3D model of the object from the plurality of 2D images further comprises:
 utilizing a 2D keypoint DLN to extract one or more keypoints from the plurality of 2D images, wherein the one or more keypoints are used to generate the annotated unstructured 3D mesh in order to generate the structured 3D model. 
 
     
     
       7. The computer-implemented method of  claim 6 , wherein constructing the structured 3D model of the object from the plurality of 2D images further comprises:
 projecting the one or more keypoints onto the scaled unstructured 3D mesh of the object to generate the annotated scaled unstructured 3D mesh, wherein the annotated scaled unstructured 3D mesh is utilized to generate the structured 3D model. 
 
     
     
       8. The computer-implemented method of  claim 6 , wherein the 2D keypoint DLN is selected from the group consisting of a stacked hourglass network and a high-resolution network (HRNet). 
     
     
       9. The computer-implemented method of  claim 1 , wherein generating the pressure map of the object from the structured 3D model and the input parameters further comprises:
 generating a density map by projecting the structured 3D model onto the surface, wherein the density map is utilized to generate the pressure map. 
 
     
     
       10. The computer-implemented method of  claim 9 ,
 wherein the pressure estimation DLN is a modified vector quantized-variational auto- encoder (VQ-VAE), 
 wherein the density map is utilized as input to the modified VQ-VAE to generate the pressure map, and 
 wherein the modified VQ-VAE is trained to generate a given pressure map from a given density map and one or more given input parameters. 
 
     
     
       11. The computer-implemented method of  claim 1 , wherein the one or more input parameters comprise at least a scale factor, and wherein the scale factor is used to scale the structured 3D model to real-world coordinates. 
     
     
       12. The computer-implemented method of  claim 1 , further comprising:
 providing instructions to manufacture a 3D product from the structured 3D model utilizing 3D measurements extracted from the structured 3D model. 
 
     
     
       13. The computer-implemented method of  claim 1 , wherein the object is a body part. 
     
     
       14. The computer-implemented method of  claim 1 , wherein the pressure estimation DLN is trained on training data comprising structured 3D models from a 3D scanner and corresponding pressure maps from an object pressure sensor. 
     
     
       15. A non-transitory storage medium for storing program code, the program code executable by a hardware processor, the program code when executed by the hardware processor causing the hardware processor to generate a pressure map of an object, the program code comprising code to:
 receive a plurality of 2-dimensional (2D) images of the object, wherein the plurality of 2D images capture the object from different angles; 
 receive one or more input parameters, the input parameters comprising at least one attribute related to the object; 
 construct a structured 3-dimensional (3D) model of the object from the plurality of 2D images; and 
 generate the pressure map of the object from the structured 3D model and the input parameters using a pressure estimation deep learning network (DLN), wherein the pressure estimation DLN is trained to generate pressure maps of objects from a given structured 3D model and given parameters of a given object, wherein the pressure map comprises a pressure profile comprising a plurality of pressure points, each pressure point indicating a level of pressure where the object is in contact with a surface, wherein the pressure estimation DLN predicts the level of pressure at the plurality of pressure points. 
 
     
     
       16. The non-transitory storage medium of  claim 15 , wherein the program code to construct the structured 3D model of the object from the plurality of 2D images comprises code to:
 generate a scaled unstructured 3D mesh of the object from the plurality of 2D images using a photogrammetry process and a scale factor, wherein the scaled unstructured 3D mesh is utilized to generate the structured 3D model. 
 
     
     
       17. The non-transitory storage medium of  claim 16 , wherein the program code to construct the structured 3D model of the object from the plurality of 2D images further comprises code to:
 generate the structured 3D model from an annotated scaled unstructured 3D mesh by morphing an annotated structured base 3D mesh to match the annotated scaled unstructured 3D mesh. 
 
     
     
       18. The non-transitory storage medium of  claim 17 , wherein the program code to construct the structured 3D model of the object from the plurality of 2D images further comprises code to:
 utilize a 3D keypoint DLN to generate the annotated scaled unstructured 3D mesh of the object from the scaled unstructured 3D mesh of the object, wherein the annotated scaled unstructured 3D mesh is utilized to generate the structured 3D model. 
 
     
     
       19. The non-transitory storage medium of  claim 17 , wherein the program code to construct the structured 3D model of the object from the plurality of 2D images further comprises code to:
 utilize a 2D keypoint DLN to extract one or more keypoints from the plurality of 2D images, wherein the one or more keypoints are used to generate the annotated unstructured 3D mesh in order to generate the structured 3D model. 
 
     
     
       20. A system comprising a hardware processor and a non-transitory storage medium for storing program code, the program code executable by the hardware processor, the program code when executed by the hardware processor causing the hardware processor to generate a pressure map of an object, the program code comprising code to:
 receive a plurality of 2-dimensional (2D) images of the object, wherein the plurality of 2D images capture the object from different angles; 
 receive one or more input parameters, the input parameters comprising at least one attribute related to the object; 
 construct a structured 3-dimensional (3D) model of the object from the plurality of 2D images; and 
 generate the pressure map of the object from the structured 3D model and the input parameters using a pressure estimation deep learning network (DLN), wherein the pressure estimation DLN is trained to generate pressure maps of objects from a given structured 3D model and given parameters of a given object, wherein the pressure map comprises a pressure profile comprising a plurality of pressure points, each pressure point indicating a level of pressure where the object is in contact with a surface, wherein the pressure estimation DLN predicts the level of pressure at the plurality of pressure points.