Systems and methods for synthetic image generation

Abstract

An image generation system is provided to: receive a 3D CAD (computer aided design) model comprising 3D model images of a target object; generate a graph data structure; based on the 3D CAD model of the target object and the graph data structure, generate a plurality of augmented CAD models of the target object comprising a plurality of data sets, each data set respectively corresponding to an associated one of a plurality of attribute classes, each data set comprising a plurality of 2D model images; input the plurality of data sets into a generative adversarial network; generate synthetic photorealistic images of the target object using the plurality of generators of the generative adversarial network, the synthetic photorealistic images including attributes in accordance with the plurality of data sets corresponding to the plurality of attribute classes; and output the synthetic photorealistic images of the target object.

The invention claimed is: 
     
       1. An image generation system for unsupervised cross-domain image generation relative to a first image domain and a second image domain, the image generation system comprising:
 a processor, and a memory storing instructions that, when executed by the processor, cause the image generation system to:
 generate a graph data structure to store in the memory, the graph data structure comprising a plurality of connected nodes, the connected nodes comprising one or more object nodes representing components and one or more part characteristic nodes representing part characteristics for each component, and one or more anomaly nodes representing potential anomalies for each part characteristic; 
 receive a 3D CAD (computer aided design) model comprising 3D model images of a target object; 
 based on the 3D CAD model of the target object and the graph data structure, generate a plurality of augmented CAD models of the target object comprising a plurality of data sets, each data set respectively corresponding to an associated one of a plurality of attribute classes, each data set comprising a plurality of 2D model images; 
 input the plurality of data sets into a generative adversarial network comprising a plurality of generators respectively corresponding to the plurality of attribute classes and a plurality of discriminators respectively corresponding to the plurality of generators; 
 generate synthetic photorealistic images of the target object using the plurality of generators, the synthetic photorealistic images including attributes in accordance with the plurality of data sets corresponding to the plurality of attribute classes; and 
 output the synthetic photorealistic images of the target object, wherein 
 
 the plurality of attribute classes includes an anomaly class; and 
 the processor is configured to generate the plurality of augmented CAD models by:
 adding one or more anomalies into the 3D CAD model augmented based on the graph data structure; and 
 rendering 2D model images of an anomaly class data set of the anomaly class using the augmented 3D CAD model. 
 
 
     
     
       2. The image generation system of  claim 1 , wherein
 the graph data structure indicates probabilities of occurrence of each anomaly represented by the one or more anomaly nodes; and 
 the 2D model images of the anomaly class data set are rendered to depict the anomalies represented by the one or more anomaly nodes based on the probabilities indicated by the graph data structure, so that proportions of the 2D model images depicting the anomalies match the probabilities indicated by the graph data structure. 
 
     
     
       3. The image generation system of  claim 1 , wherein
 each of the one or more anomaly nodes of the graph data structure is associated with anomaly parameters for depicting the anomalies on the augmented CAD models; 
 the one or more anomalies are added into the 3D CAD model augmented based on the anomaly parameters; and 
 the 2D model images of the anomaly class data set are rendered to depict the anomalies represented by the one or more anomaly nodes based on the anomaly parameters included in the graph data structure. 
 
     
     
       4. The image generation system of  claim 1 , wherein the graph data structure is generated based on manufacturing inspection data of the target object. 
     
     
       5. The image generation system of  claim 1 , wherein the plurality of attribute classes further include at least one of a material textures class, an environmental conditions class, or a perspective views class. 
     
     
       6. The image generation system of  claim 5 , wherein
 one of the attribute classes is the material textures class; 
 one of the plurality of generators is a material textures class generator including:
 a first image generator configured to generate synthetic images having a similar appearance to one or more real photorealistic images in the second image domain while including a semantic content of one or more 2D model images depicting at least one of materials or textures in the first image domain in accordance with a material textures class data set in the augmented CAD models of the target object; 
 a second image generator configured to generate synthetic 2D model images having a similar appearance to one or more of the 2D model images depicting at least one of materials or textures in the first image domain while including semantic content of the one or more real photorealistic images in the second image domain; and 
 
 one of the plurality of discriminators is a material textures class discriminator including:
 a first image discriminator configured to discriminate the real photorealistic images in the second image domain against the synthetic photorealistic images generated by the first image generator; and 
 a second image discriminator configured to discriminate real 2D model images depicting at least one of materials or textures in the first image domain against the synthetic 2D model images depicting at least one of materials or textures generated by the second image generator. 
 
 
     
     
       7. The image generation system of  claim 5 , wherein the processor is configured to generate the plurality of augmented CAD models by:
 adding one or more material textures into the 3D CAD model augmented in accordance with the graph data structure; and 
 rendering 2D model images of a material textures class data set of the material textures class using the augmented 3D CAD model. 
 
     
     
       8. The image generation system of  claim 5 , wherein
 one of the attribute classes is the environmental conditions class; 
 one of the plurality of generators is an environmental conditions class generator including:
 a first image generator configured to generate synthetic photorealistic images having a similar appearance to one or more real photorealistic images in the second image domain while including a semantic content of one or more 2D model images depicting environmental conditions in the first image domain in accordance with an environmental conditions class data set in the augmented CAD models of the target object; 
 a second image generator configured to generate synthetic 2D model images having a similar appearance to one or more of the 2D model images depicting environmental conditions in the first image domain while including semantic content of the one or more real photorealistic images in the second image domain; and 
 
 one of the plurality of discriminators is an environmental conditions class discriminator including:
 a first image discriminator configured to discriminate the real photorealistic images in the second image domain against the synthetic photorealistic images generated by the first image generator; and 
 a second image discriminator configured to discriminate real 2D model images depicting environmental conditions in the first image domain against the synthetic 2D model images depicting environmental conditions generated by the second image generator. 
 
 
     
     
       9. The image generation system of  claim 5 , wherein the processor is configured to generate the plurality of augmented CAD models by:
 adding one or more environmental conditions into the 3D CAD model augmented in accordance with the graph data structure; and 
 rendering 2D model images of an environmental conditions class data set of the environmental conditions class using the augmented 3D CAD model. 
 
     
     
       10. The image generation system of  claim 5 , wherein
 one of the attribute classes is the perspective views class; 
 one of the plurality of generators is a perspective views class generator including:
 a first image generator configured to generate synthetic photorealistic images having a similar appearance to one or more real photorealistic images in the second image domain while including a semantic content of one or more 2D model images depicting different perspective views of the target object in the first image domain in accordance with a perspective views class data set in the augmented CAD models of the target object; 
 a second image generator configured to generate synthetic 2D model images having a similar appearance to one or more of the 2D model images depicting different perspective views of the target object in the first image domain while including semantic content of the one or more real photorealistic images in the second image domain; and 
 
 one of the plurality of discriminators is a perspective views class discriminator including:
 a first image discriminator configured to discriminate the real photorealistic images in the second image domain against the synthetic photorealistic images generated by the first image generator; and 
 a second image discriminator configured to discriminate real 2D model images depicting different perspective views of the target object in the first image domain against the synthetic 2D model images depicting different perspective views of the target object generated by the second image generator. 
 
 
     
     
       11. The image generation system of  claim 5 , wherein the processor is configured to generate the plurality of augmented CAD models by:
 taking 2D model images of the 3D CAD model from one or more perspective views augmented in accordance with the graph data structure; and 
 rendering 2D model images of a perspective views class data set of the perspective views class using the augmented 3D CAD model. 
 
     
     
       12. The image generation system of  claim 1 , wherein the processor is configured to generate at least one of the plurality of data sets as a plurality of 2D model images comprising a plurality of 2D pixelated images. 
     
     
       13. An image generation method for unsupervised cross-domain image generation relative to a first image domain and a second image domain, the image generation method comprising:
 generating and storing a graph data structure in memory, the graph data structure comprising a plurality of connected nodes, the connected nodes comprising one or more object nodes representing components and one or more part characteristic nodes representing part characteristics for each component, and one or more anomaly nodes representing potential anomalies for each part characteristic; 
 receiving a 3D CAD (computer aided design) model comprising 3D model images of a target object; 
 based on the 3D CAD model of the target object and the graph data structure, generating a plurality of augmented CAD models of the target object comprising a plurality of data sets, each data set respectively corresponding to an associated one of a plurality of attribute classes, each data set comprising a plurality of 2D model images; 
 inputting the plurality of data sets into a generative adversarial network comprising a plurality of generators respectively corresponding to the plurality of attribute classes and a plurality of discriminators respectively corresponding to the plurality of generators; 
 generating synthetic photorealistic images of the target object using the plurality of generators, the synthetic photorealistic images including attributes in accordance with the plurality of data sets corresponding to the plurality of attribute classes; and 
 outputting the synthetic photorealistic images of the target object, wherein 
 the plurality of attribute classes includes an anomaly class; and 
 the plurality of augmented CAD models is generated by:
 adding one or more anomalies into the 3D CAD model augmented based on the graph data structure; and 
 rendering 2D model images of an anomaly class data set of the anomaly class using the augmented 3D CAD model. 
 
 
     
     
       14. The image generation method of  claim 13 , wherein
 the graph data structure indicates probabilities of occurrence of each anomaly represented by the one or more anomaly nodes; and 
 the 2D model images of the anomaly class data set are rendered to depict the anomalies represented by the one or more anomaly nodes based on the probabilities indicated by the graph data structure, so that proportions of the 2D model images depicting the anomalies match the probabilities indicated by the graph data structure. 
 
     
     
       15. The image generation method of  claim 13 , wherein
 each of the one or more anomaly nodes of the graph data structure is associated with anomaly parameters for depicting the anomalies on the augmented CAD models; 
 the one or more anomalies are added into the 3D CAD model augmented based on the anomaly parameters; 
 the 2D model images of the anomaly class data set are rendered to depict the anomalies represented by the one or more anomaly nodes based on the anomaly parameters included in the graph data structure. 
 
     
     
       16. The image generation method of  claim 13 , wherein the plurality of attribute classes further include at least one of a material textures class, an environmental conditions class, or a perspective views class. 
     
     
       17. The image generation method of  claim 13 , wherein the graph data structure is generated based on manufacturing inspection data of the target object. 
     
     
       18. The image generation method of  claim 16 , wherein the plurality of augmented CAD models are generated by:
 adding one or more material textures into the 3D CAD model augmented in accordance with the graph data structure; and 
 rendering 2D model images of a material textures class data set of the material textures class using the augmented 3D CAD model. 
 
     
     
       19. The image generation method of  claim 13 , wherein at least one of the plurality of data sets is generated as a plurality of 2D model images comprising a plurality of 2D pixelated images. 
     
     
       20. An image generation system for unsupervised cross-domain image generation relative to a first image domain and a second image domain, the image generation system comprising:
 a processor, and a memory storing instructions that, when executed by the processor, cause the system to:
 receive a 3D CAD (computer aided design) model comprising 3D model images of a target object; 
 receiving manufacturing inspection data of the target object; 
 generate a graph data structure based on the manufacturing inspection data of the target object; 
 storing the graph data structure in the memory, the graph data structure comprising a plurality of connected nodes, the connected nodes comprising one or more object nodes representing components and one or more part characteristic nodes representing part characteristics for each component, and one or more anomaly nodes representing potential anomalies for each part characteristic; 
 based on the 3D CAD model of the target object and the graph data structure, generate a plurality of augmented CAD models of the target object comprising a plurality of data sets, each data set respectively corresponding to an associated one of a plurality of attribute classes, each data set comprising a plurality of 2D model images; 
 input the plurality of data sets into a generative adversarial network comprising a plurality of generators respectively corresponding to the plurality of attribute classes and a plurality of discriminators respectively corresponding to the plurality of generators; 
 generate synthetic photorealistic images of the target object using the plurality of generators, the synthetic photorealistic images including attributes in accordance with the plurality of data sets corresponding to the plurality of attribute classes; and 
 inputting the synthetic photorealistic images of the target object into an anomaly detection artificial intelligence model to train the anomaly detection artificial intelligence model to detect anomalies on the target object, wherein 
 
 the plurality of attribute classes includes an anomaly class; and 
 the plurality of augmented CAD models are generated by:
 adding one or more anomalies into the 3D CAD model augmented based on the graph data structure indicating probabilities of occurrence of each anomaly represented by the one or more anomaly nodes; and 
 rendering 2D model images of an anomaly class data set of the anomaly class using the augmented 3D CAD model to depict the anomalies represented by the one or more anomaly nodes based on the probabilities indicated by the graph data structure, so that proportions of the 2D model images depicting the anomalies match the probabilities indicated by the graph data structure.