Three-dimensional modeling toolkit

Abstract

A system includes one or more hardware processors and memory storing instructions that, when executed by the one or more hardware processors, causes the system to access image data depicting an object including generating the image data via a camera of a mobile device of the user where the object is a portion of the user, identify a set of features, assign labels to regions of the image data based on the set of features including providing the set of features identified from each respective region of the regions as input into a model and where the model outputs a label for the respective region and where the labels comprise one or more semantic labels or classifications that correspond with the set of features, and generate a three-dimensional (3D) model of the object based on the labels assigned to the regions of the image data.

What is claimed is: 
     
         1 . A system comprising:
 one or more hardware processors; and   memory storing instructions that, when executed by the one or more hardware processors, causes the system to perform operations comprising:   accessing image data depicting an object, wherein accessing the image data includes generating the image data via a camera of a mobile device of the user, and wherein the object is a portion of the user;   identifying a set of features from the image data;   assigning labels to regions of the image data based on the set of features, wherein assigning labels to the regions of the image data comprises providing the set of features identified from each respective region of the regions as input into a model, wherein the model outputs a label for the respective region, and wherein the labels comprise one or more semantic labels or classifications that correspond with the set of features; and   generating a three-dimensional (3D) model of the object based on the labels assigned to the regions of the image data.   
     
     
         2 . The system of  claim 1 , wherein the one or more semantic labels or classifications indicate one or more regions within the image data to scan. 
     
     
         3 . The system of  claim 1 , wherein accessing the image data includes:
 activating the camera associated with the mobile device of the user; and   generating the image data via the camera of the mobile device, wherein the image data corresponds to a first data stream.   
     
     
         4 . The system of  claim 3 , wherein generating the 3D model based on the labels includes:
 generating the 3D model based on at least one of 2D-2D correspondences of the image data or 2D-3D correspondences of the image data, wherein the 2D-2D correspondences or the 2D-3D correspondences includes a plurality of distances of points in the image data depicting the object to the camera of the mobile device, wherein the 2D-2D correspondences or the 2D-3D correspondences corresponds to a second data stream.   
     
     
         5 . The system of  claim 1 , wherein the 3D model is a reconstruction of the object, and wherein the reconstruction is performed using one or more of a polygon mesh model, a triangle mesh model, a non-uniform rational basis spline (NURBS) surface model, or a CAD model. 
     
     
         6 . The system of  claim 1 , wherein the image data comprises a depiction of the object, and wherein the 3D model comprises a representation of surface features of the object depicted by the image data, and wherein the representation of surface features correspond to at least a portion of the user's head or face. 
     
     
         7 . The system of  claim 6 , wherein the image data comprises a plurality of objects, and the operations further comprise:
 filtering the plurality of objects of the user's head or face corresponding to a plurality of size values based on a position of the 3D model relative to a point cloud; and   presenting, via an interface, the filtered 3D model on the mobile device of the user.   
     
     
         8 . The system of  claim 1 , wherein the 3D model includes unlabeled regions and labeled regions, and wherein the labeled regions correspond to a different color or pattern from the labeled regions. 
     
     
         9 . A method comprising:
 accessing, by a processing circuit, image data depicting an object, wherein accessing the image data includes generating the image data via a camera of a mobile device of the user, and wherein the object is a portion of the user;   identifying, by the processing circuit, a set of features from the image data;   assigning, by the processing circuit, labels to regions of the image data based on the set of features, wherein assigning labels to the regions of the image data comprises providing the set of features identified from each respective region of the regions as input into a model, wherein the model outputs a label for the respective region, and wherein the labels comprise one or more semantic labels or classifications that correspond with the set of features; and   generating, by the processing circuit, a three-dimensional (3D) model of the object based on the labels assigned to the regions of the image data.   
     
     
         10 . The method of  claim 9 , wherein the one or more semantic labels or classifications indicate one or more regions within the image data to scan. 
     
     
         11 . The method of  claim 9 , wherein accessing the image data includes:
 activating the camera associated with the mobile device of the user; and   generating the image data via the camera of the mobile device, wherein the image data corresponds to a first data stream.   
     
     
         12 . The method of  claim 11 , wherein generating the 3D model based on the labels includes:
 generating the 3D model based on at least one of 2D-2D correspondences of the image data or 2D-3D correspondences of the image data, wherein the 2D-2D correspondences or the 2D-3D correspondences includes a plurality of distances of points in the image data depicting the object to the camera of the mobile device, wherein the 2D-2D correspondences or the 2D-3D correspondences corresponds to a second data stream.   
     
     
         13 . The method of  claim 9 , wherein the 3D model is a reconstruction of the object, and wherein the reconstruction is performed using one or more of a polygon mesh model, a triangle mesh model, a non-uniform rational basis spline (NURBS) surface model, or a CAD model. 
     
     
         14 . The method of  claim 9 , wherein the image data comprises a depiction of the object, and wherein the 3D model comprises a representation of surface features of the object depicted by the image data, and wherein the representation of surface features correspond to at least a portion of the user's head or face. 
     
     
         15 . The method of  claim 14 , wherein the image data comprises a plurality of objects, and the method further comprising:
 filtering the plurality of objects of the user's head or face corresponding to a plurality of size values based on a position of the 3D model relative to a point cloud; and   presenting, via an interface, the filtered 3D model on the mobile device of the user.   
     
     
         16 . The method of  claim 9 , wherein the 3D model includes unlabeled regions and labeled regions, and wherein the labeled regions correspond to a different color or pattern from the labeled regions. 
     
     
         17 . A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computer system, cause the computer system to perform operations comprising:
 accessing image data depicting an object, wherein accessing the image data includes generating the image data via a camera of a mobile device of the user, and wherein the object is a portion of the user;   identifying a set of features from the image data;   assigning labels to regions of the image data based on the set of features, wherein assigning labels to the regions of the image data comprises providing the set of features identified from each respective region of the regions as input into a model, wherein the model outputs a label for the respective region, and wherein the labels comprise one or more semantic labels or classifications that correspond with the set of features; and   generating a three-dimensional (3D) model of the object based on the labels assigned to the regions of the image data.   
     
     
         18 . The non-transitory computer-readable storage medium of  claim 17 , wherein the one or more semantic labels or classifications indicate one or more regions within the image data to scan. 
     
     
         19 . The non-transitory computer-readable storage medium of  claim 17 , wherein the operations further comprise:
 activating the camera associated with the mobile device of the user;   generating the image data via the camera of the mobile device, wherein the image data corresponds to a first data stream; and   generating the 3D model based on at least one of 2D-2D correspondences of the image data or 2D-3D correspondences of the image data, wherein the 2D-2D correspondences or the 2D-3D correspondences includes a plurality of distances of points in the image data depicting the object to the camera of the mobile device, wherein the 2D-2D correspondences or the 2D-3D correspondences corresponds to a second data stream.   
     
     
         20 . The non-transitory computer-readable storage medium of  claim 17 , wherein the image data comprises a depiction of the object, and wherein the 3D model comprises a representation of surface features of the object depicted by the image data, and wherein the representation of surface features correspond to at least a portion of the user's head or face.