Machine learning models in an artificial intelligence infrastructure

Abstract

Improving machine learning models in an artificial intelligence infrastructure includes: storing, within one or more storage systems of an artificial intelligence infrastructure, information describing a dataset and one or more transformations applied to the dataset resulting in a transformed dataset; and storing, within the one or more storage systems, information describing only portions of previous versions of a machine learning model that differ from a current version of the machine learning model, wherein the previous versions used the transformed dataset as input during one or more prior executions by the artificial intelligence infrastructure.

What is claimed is: 
     
       1. A method comprising:
 storing, within one or more storage systems of an artificial intelligence infrastructure, information describing a dataset and one or more transformations applied to the dataset resulting in a transformed dataset; 
 identifying, by the artificial intelligence infrastructure, previous versions of a machine learning model that used the transformed dataset as input during one or more prior executions by the artificial intelligence infrastructure; 
 storing, within the one or more storage systems, information describing only differences between the previous versions of the machine learning model and a current version of the machine learning model; 
 retaining one or more first portions of the transformed dataset associated with the current version of the machine learning model at a first storage tier; and 
 moving one or more second portions of the transformed dataset associated with the previous versions of the machine learning model to a second storage tier. 
 
     
     
       2. The method of  claim 1  wherein the storing, within the one or more storage systems of the artificial intelligence infrastructure, the information describing the dataset and the one or more transformations applied to the dataset resulting in the transformed dataset further comprises:
 generating, by the artificial intelligence infrastructure applying a predetermined hash function to the dataset, the one or more transformations applied to the dataset, and the transformed dataset, a hash value; and 
 storing, within the one or more storage systems, the hash value. 
 
     
     
       3. The method of  claim 1  wherein the storing, within the one or more storage systems, information describing only differences between previous versions of the machine learning model and a current version of the machine learning model further comprises:
 generating, by the artificial intelligence infrastructure applying a predetermined hash function to the previous versions of the machine learning model and the transformed dataset, a hash value; and 
 storing, within the one or more storage systems, the hash value. 
 
     
     
       4. The method of  claim 1  further comprising:
 identifying differences between the current version of the machine learning model and the previous versions of the machine learning model. 
 
     
     
       5. The method of  claim 1  further comprising:
 determining, by the artificial intelligence infrastructure, whether data related to one or more of the previous versions of the machine learning model should be tiered off of the one or more storage systems; and 
 responsive to determining that the data related to the one or more of the previous versions of the machine learning model should be tiered off of the one or more storage systems:
 storing the data related to the one or more of the previous versions of the machine learning model in lower-tier storage; and 
 removing, from the one or more storage systems, the data related to the one or more of the previous versions of the machine learning model. 
 
 
     
     
       6. The method of  claim 1  further comprising identifying, from amongst the previous versions and the current version of the machine learning model, a preferred version of the machine learning model. 
     
     
       7. The method of  claim 1  further comprising tracking an improvement of a particular version of the machine learning model over time. 
     
     
       8. An artificial intelligence infrastructure comprising:
 one or more storage systems; 
 one or more graphical processing unit (‘GPU’) servers; and 
 a processing device, operatively coupled to the one or more storage systems and one or more GPU servers, the processing device configured to: 
 store, within the one or more storage systems, information describing a dataset and one or more transformations applied to the dataset resulting in a transformed dataset; 
 obtain, by the artificial intelligence infrastructure, identifiers for previous versions of a machine learning model that used the transformed dataset as input during one or more prior executions by the artificial intelligence infrastructure; 
 store, within the one or more storage systems, information describing only differences between the previous versions of the machine learning model and a current version of the machine learning model; 
 retain one or more first portions of the transformed dataset associated with the current version of the machine learning model at a first storage tier; and 
 move one or more second portions of the transformed dataset associated with the previous versions of the machine learning model to a second storage tier. 
 
     
     
       9. The artificial intelligence infrastructure of  claim 8  wherein to store, within the one or more storage systems, the information describing the dataset and the one or more transformations applied to the dataset resulting in the transformed dataset the processing device is further configured to:
 generate, by the artificial intelligence infrastructure applying a predetermined hash function to the one or more transformations applied to the dataset and the transformed dataset, a hash value; and 
 store, within the one or more storage systems, the hash value. 
 
     
     
       10. The artificial intelligence infrastructure of  claim 8  wherein to store, within the one or more storage systems, information describing only differences between previous versions of the machine learning model and a current version of the machine learning model the processing device is further configured to:
 generate, by the artificial intelligence infrastructure applying a predetermined hash function to the previous versions of the machine learning model, a hash value; and 
 store, within the one or more storage systems, the hash value. 
 
     
     
       11. The artificial intelligence infrastructure of  claim 8  wherein the processing device is further configured to:
 identity, by a unified management plane, differences between the current version of the machine learning model and the previous versions of the machine learning model. 
 
     
     
       12. The artificial intelligence infrastructure of  claim 8  wherein the processing device is further configured to:
 determining, by the artificial intelligence infrastructure, whether data related to one or more of the previous versions of the machine learning model should be tiered off of the one or more storage systems; and 
 responsive to determining that the data related to the one or more of the previous versions of the machine learning model should be tiered off of the one or more storage systems:
 storing the data related to the one or more of the previous versions of the machine learning model in lower-tier storage; and 
 removing, from the one or more storage systems, the data related to the one or more of the previous versions of the machine learning model. 
 
 
     
     
       13. The artificial intelligence infrastructure of  claim 8  wherein the processing device is further configured to:
 identify, from amongst the previous versions and the current version of the machine learning model, a preferred version of the machine learning model. 
 
     
     
       14. The artificial intelligence infrastructure of  claim 8  wherein wherein the processing device is further configured to tracking an improvement of a particular version of the machine learning model over time. 
     
     
       15. An apparatus comprising:
 a memory; and 
 a processing device, operatively coupled with the memory, the processing device configured to: 
 store, within one or more storage systems of an artificial intelligence infrastructure, information describing a dataset and one or more transformations applied to the dataset resulting in a transformed dataset; 
 obtain, by the artificial intelligence infrastructure, identifiers for previous versions of a machine learning model that used the transformed dataset as input during one or more prior executions by the artificial intelligence infrastructure; 
 store, within the one or more storage systems, information describing only differences between the previous versions of the machine learning model and a current version of the machine learning model; 
 retain one or more first portions of the transformed dataset associated with the current version of the machine learning model at a first storage tier; and 
 move one or more second portions of the transformed dataset associated with the previous versions of the machine learning model to a second storage tier. 
 
     
     
       16. The apparatus of  claim 15  wherein the processing device is further configured to:
 identify, by a unified management plane, differences between the current version of the machine learning model and the previous versions of the machine learning model. 
 
     
     
       17. The apparatus of  claim 15  wherein the processing device is further configured to:
 determining, by the artificial intelligence infrastructure, whether data related to one or more of the previous versions of the machine learning model should be tiered off of the one or more storage systems; and 
 responsive to determining that the data related to the one or more of the previous versions of the machine learning model should be tiered off of the one or more storage systems:
 storing the data related to the one or more of the previous versions of the machine learning model in lower-tier storage; and 
 removing, from the one or more storage systems, the data related to the one or more of the previous versions of the machine learning model. 
 
 
     
     
       18. The apparatus of  claim 15  wherein the processing device is further configured to:
 identify, from amongst the previous versions and the current version of the machine learning model, a preferred version of the machine learning model. 
 
     
     
       19. The apparatus of  claim 15  wherein the processing device is further configured to:
 track an improvement of a particular version of the machine learning model over time. 
 
     
     
       20. The apparatus of  claim 15  wherein the processing device is further configured to:
 generate, by the artificial intelligence infrastructure applying a predetermined hash function to the dataset, and the one or more transformations applied to the dataset, a hash value; and 
 store, within the one or more storage systems, the hash value.