Synthetic infrared data for image classification systems and methods

Abstract

An image classification system comprises a neural network trained on a synthetic infrared training dataset, including synthetic infrared images of objects rendered from a virtually represented infrared sensor in a virtual three-dimensional scene, the synthetic infrared images being generated using infrared radiation signatures of virtual objects in the virtual three-dimensional scene and an infrared response model of the virtually represented infrared sensor. A system for generating synthetic infrared training data comprises a three-dimensional scene modeling system operable to generate three-dimensional scenes comprising a plurality of objects, each object having an infrared radiation model, and an infrared sensor modeling system operable to model an imaging response for an infrared sensor virtually represented in the three-dimensional scene.

What is claimed is: 
     
       1. A system comprising:
 one or more processors and memory devices configured to: 
 receive an infrared image captured from an infrared sensor; and 
 implement an image classification system operable to classify the infrared image, the image classification system comprising a neural network trained on synthetic infrared training data comprised of synthetic infrared images rendered by applying infrared radiation models to corresponding virtual objects in a virtual three-dimensional scene and constructing a synthetic infrared image of the virtual three-dimensional scene in accordance with an infrared sensor response model; 
 wherein at least one virtual three-dimensional scene is generated in response to a neural network validation process; and 
 wherein at least one synthetic infrared image of the at least one virtual three-dimensional scene is added to the synthetic infrared training data resulting in a reduction of a validation error from the neural network validation process. 
 
     
     
       2. The system of  claim 1 , wherein the infrared sensor comprises a multi spectral dynamic image sensor having infrared sensor components and visible light sensor components; wherein the infrared image is enhanced by a visible light image generated from the visible light sensor components; and wherein the infrared sensor response model produces an imaging response for a virtual multi spectral dynamic image sensor. 
     
     
       3. The system of  claim 1 , further comprising:
 a database storing the synthetic infrared training data. 
 
     
     
       4. The system of  claim 1 , wherein the neural network comprises a convolutional neural network. 
     
     
       5. The system of  claim 1 , wherein the one or more processors and memory devices are further configured to implement an infrared training data generation system comprising:
 an object database comprising a plurality of three-dimensional objects; 
 a three-dimensional scene modeling system operable to generate the virtual three-dimensional scene comprising a subset of the plurality of three-dimensional objects; 
 an infrared radiation model database comprising infrared radiation models corresponding to the three-dimensional objects in the object database; 
 an infrared sensor modeling system operable to model an imaging response for an infrared sensor virtually represented in the virtual three-dimensional scene; and 
 a synthetic infrared imaging system operable to render the synthetic infrared images as received by the virtually represented infrared sensor in the virtual three-dimensional scene, each synthetic infrared image being generated using the infrared radiation models of the three-dimensional objects in the virtual three-dimensional scene and the model of the infrared sensor; 
 wherein the at least one synthetic infrared image of the at least one virtual three-dimensional scene added to the synthetic infrared training data is generated by increasing resolution to help distinguish between objects, thereby adjusting the synthetic infrared training data to improve accuracy. 
 
     
     
       6. The system of  claim 5 , wherein the one or more processors and memory devices are further configured to implement:
 an environmental component operable to apply environmental effects to the virtual three-dimensional scene, including at least one of sun, lighting, and weather elements; and wherein: 
 the synthetic infrared imaging system is operable to render an image including virtual motion of one or more objects in the virtual three-dimensional scene; 
 the synthetic infrared image is stored in a training set database; 
 the virtual three-dimensional scene is generated in response to a neural network validation process, and wherein an accuracy of at least one of the infrared radiation models is adjusted in response to a detected validation error; and 
 the synthetic infrared imaging system is further operable to add noise to a subset of the synthetic infrared images to account for sensor variations between infrared imaging devices. 
 
     
     
       7. A method comprising:
 capturing an infrared image of a scene; and 
 classifying at least one object in the captured infrared image by feeding the infrared image to a neural network trained on a synthetic infrared training dataset comprising a plurality of synthetically generated infrared images; wherein each of the plurality of synthetically generated infrared images is rendered as a response received by a virtual infrared sensor in a virtual three-dimensional scene comprising a plurality of virtual three-dimensional objects; and 
 determining a validation error in the classifying at least one object in the infrared image; and 
 generating a new virtual three-dimensional scene in response to the validation error; 
 rendering a new synthetic infrared image of the new virtual three-dimensional scene; and 
 adding the new synthetic infrared image to the training dataset to produce an updated training dataset to reduce the validation error of a neural network training on the updated training dataset. 
 
     
     
       8. The method of  claim 7 , further comprising capturing a visible spectrum image of the scene, wherein the infrared image is enhanced by the captured visible spectrum image. 
     
     
       9. The method of  claim 8 , further comprising applying a virtual visible light sensor to the virtual three-dimensional scene to provide a synthetic visual image, and wherein the synthetic infrared training dataset includes synthetic images enhanced by visible light. 
     
     
       10. The method of  claim 7 , further comprising generating the synthetic infrared training dataset, wherein generating the synthetic infrared training dataset comprises:
 defining a plurality of three-dimensional objects; 
 defining a plurality of infrared radiation models, wherein each of the plurality of three-dimensional objects is associated with at least one infrared radiation model; 
 defining an infrared sensor model to virtually represent an imaging response of an infrared sensor; 
 building a plurality of virtual three-dimensional scenes, each scene comprising one or more of the plurality of three-dimensional objects; and 
 generating a synthetic infrared image of each of a plurality of virtual three-dimensional scenes by applying a corresponding infrared radiation model to each three-dimensional object in the virtual three-dimensional scene, and determining an imaging sensor response to the scene using the infrared sensor model. 
 
     
     
       11. The method of  claim 10 , wherein applying the corresponding infrared radiation model comprises applying a motion characteristic to at least one of the three-dimensional objects in the virtual three-dimensional scene. 
     
     
       12. The method of  claim 10 , further comprising:
 storing each image in the synthetic infrared training dataset with corresponding annotations describing the scene; 
 validating the synthetic infrared training dataset, and adjusting an accuracy of at least one infrared radiation model in response to a detected validation error; and 
 adding noise to a subset of the synthetically generated infrared images to account for noise generated by the infrared sensor. 
 
     
     
       13. A system comprising:
 one or more processors and memory devices configured to implement: 
 an object database comprising a plurality of three-dimensional objects; 
 a three-dimensional scene modeling system operable to generate one or more virtual three-dimensional scenes comprising a subset of the plurality of three-dimensional objects; 
 an infrared radiation model database comprising infrared radiation models corresponding to the objects in the object database; 
 an infrared sensor modeling system operable to model an imaging response for an infrared sensor virtually represented in the virtual three-dimensional scene; and 
 a synthetic infrared imaging system operable to render a synthetic infrared image received from the virtually represented infrared sensor in the three-dimensional scene, the synthetic infrared image being generated using the infrared radiation models of the three-dimensional objects in the generated virtual three-dimensional scene and the virtually represented model of the infrared sensor. 
 
     
     
       14. The system of  claim 13 , wherein the one or more processors and memories is further configured to implement:
 an environmental component operable to apply environmental effects to the virtual three-dimensional scene, including at least one of sun, lighting, and weather elements; and 
 wherein the synthetic infrared imaging system is operable to render an image including virtual motion of one or more objects in the virtual three-dimensional scene. 
 
     
     
       15. The system of  claim 14 , wherein:
 the synthetic infrared image is stored in a training set database; 
 the virtual three-dimensional scene is generated to train a neural network for object classification; and 
 the virtual three-dimensional scene is generated in response to a neural network validation process, 
 in response to a validation error being detected an object in the scene is rendered at a higher resolution to generate a high resolution synthetic infrared image, and 
 the high resolution synthetic infrared image is added to the training set database, resulting in a reduction of a validation error from the neural network validation process. 
 
     
     
       16. The system of  claim 14 , wherein a plurality of synthetic infrared images are generated using the infrared sensor model, and wherein the synthetic infrared imaging system is further operable to add noise to a subset of the synthetic infrared images to account for sensor variations between infrared imaging devices. 
     
     
       17. A method of using the system of  claim 13  comprising:
 defining parameters of a training dataset including at least one object for classification, at least one infrared sensor for infrared image capture, and at least one use environment; 
 building a three-dimensional virtual environment in accordance with the parameters; 
 generating imaging scenarios including randomized object placement within the three-dimensional virtual environment; and 
 for each imaging scenario, generating a corresponding synthetic infrared image by applying infrared radiation response models to objects in the three-dimensional virtual environment and simulating an infrared sensor response. 
 
     
     
       18. The method of  claim 17 , further comprising:
 storing each synthetic infrared image in a database with corresponding annotations describing the corresponding imaging scenario. 
 
     
     
       19. The method of  claim 18 , further comprising:
 training a neural network using the stored synthetic infrared images; 
 validating the training using infrared images captured by an infrared sensor to detect classification errors; and 
 updating the parameters in response to detected classification errors to improve classification accuracy. 
 
     
     
       20. A database of synthetic images generated according to the method of  claim 17 .