Utilizing a protected server environment to protect data used to train a machine learning system

Abstract

Media items associated with status values are stored using a server in a protected environment. A device outside the protected environment requests sending media items to a client device that is also outside. Using machine learning systems that can be trained using attribute values associated with personal data records to output the existence of a status value, the server computer trains a particular machine learning system in the protected environment only if specified data meets specified criteria, then sends the resulting trained ML system to the requesting device. That device evaluates the trained ML system to determine which media items to deliver to the client device in the manner set forth in the claims.

What is claimed is: 
     
       1. A computer-implemented method comprising:
 storing, using a server computer executing within a protected environment, a plurality of media items, each of the media items corresponding to one of a plurality of different status values; 
 receiving, from a requesting computing device that is outside the protected environment, a request to send certain media items outside the protected environment to a client computing device; 
 computing, using a plurality of machine learning systems executed by the server computer, each of the machine learning systems having been trained with one of the plurality of status values as an output, a plurality of likelihood values associated with a particular status value for the client computing device, each of the machine learning systems having been trained at least in part by receiving, by the server computer executing within the protected environment, instructions to generate and train a particular machine learning system, using attribute values associated with personal data records as inputs, and an existence or a non-existence of a one of the plurality of different status values as outputs, the server computer storing first data comprising a plurality of attribute values for a plurality of the personal data records and second data indicating, for each personal data record of the plurality of personal data records, whether the personal data record has the status value, the server computer being configured to train the particular machine learning system in the protected environment only if the first data and the second data satisfy a first criterion and being configured to send the particular machine learning system to the requesting computing device only if the particular machine learning system satisfies a second criterion; 
 identifying a particular status value, among the plurality of status values, having a highest likelihood value; 
 selecting a specific set of media items at least partly based on the identified particular status value having the highest likelihood value, in a number indicated by the request to send certain media items outside the protected environment to the client computing device; and 
 sending, from the server computer to the client computing device, the specific set of media items that have been selected. 
 
     
     
       2. The method of  claim 1 , further comprising the server computer using the highest likelihood value associated with the particular status value to dynamically price sending media items to the client computing device by determining a charged price by discounting a standard price by an amount corresponding to a percentage value. 
     
     
       3. The method of  claim 1 , further comprising the server computer requesting attribute data from an outside attribute database based on information received from the client computing device. 
     
     
       4. The method of  claim 1 , further comprising:
 receiving, from the requesting computing device that is outside the protected environment, particular attributes for the client computing device; and 
 determining, based on the particular attributes, whether to serve a particular media item to the client computing device. 
 
     
     
       5. The method of  claim 1 , further comprising the server computer storing attribute values for a plurality of different client computing devices in an attribute database in the protected environment. 
     
     
       6. The method of  claim 1 , the first criterion being a minimum number of instances in the second data of a particular personal data record having the status value. 
     
     
       7. The method of  claim 1 , the second criterion being a maximum fraction of population at risk. 
     
     
       8. The method of  claim 7 , further comprising computing the maximum fraction of population at risk as a quotient of a number of instances in the subset of the first data of a patient having the status value and a number of positive predictions of the status value from applying the particular machine learning system to each of the plurality of personal data records in the first data. 
     
     
       9. The method of  claim 1 , further comprising:
 training the particular machine learning system with a first set of parameters; and 
 determining that the particular machine learning system does not satisfy the second criterion and, in response, training the particular machine learning system using a second set of parameters. 
 
     
     
       10. The method of  claim 1 , the status being one of a particular medical diagnosis or a particular prescription. 
     
     
       11. A computer system comprising:
 one or more processors; and 
 one or more computer-readable non-transitory storage media coupled to one or more of the processors and storing instructions operable when executed by one or more of the processors to cause the system to perform a method comprising: 
 storing, using a server computer executing within a protected environment, a plurality of media items, each of the media items corresponding to one of a plurality of different status values; 
 receiving, from a requesting computing device that is outside the protected environment, a request to send certain media items outside the protected environment to a client computing device; 
 computing, using a plurality of machine learning systems executed by the server computer, each of the machine learning systems having been trained with one of the plurality of status values as an output, a plurality of likelihood values associated with a particular status value for the client computing device, each of the machine learning systems having been trained at least in part by receiving, by the server computer executing within the protected environment, instructions to generate and train a particular machine learning system, using attribute values associated with personal data records as inputs, and an existence or a non-existence of a one of the plurality of different status values as outputs, the server computer storing first data comprising a plurality of attribute values for a plurality of the personal data records and second data indicating, for each personal data record of the plurality of personal data records, whether the personal data record has the status value, the server computer being configured to train the particular machine learning system in the protected environment only if the first data and the second data satisfy a first criterion and being configured to send the particular machine learning system to the requesting computing device only if the particular machine learning system satisfies a second criterion; 
 identifying a particular status value, among the plurality of status values, having a highest likelihood value; 
 selecting a specific set of media items at least partly based on the identified particular status value having the highest likelihood value, in a number indicated by the request to send certain media items outside the protected environment to the client computing device; and 
 sending, from the server computer to the client computing device, the specific set of media items that have been selected. 
 
     
     
       12. The system of  claim 11 , the storage media further comprising instructions which when executed by one or more of the processors cause the system to perform using the highest likelihood value associated with the particular status value to dynamically price sending media items to the client computing device by determining a charged price by discounting a standard price by an amount corresponding to a percentage value. 
     
     
       13. The system of  claim 11 , the storage media further comprising instructions which when executed by one or more of the processors cause the system to perform requesting attribute data from an outside attribute database based on information received from the client computing device. 
     
     
       14. The system of  claim 11 , the storage media further comprising instructions which when executed by one or more of the processors cause the system to perform:
 receiving, from the requesting computing device that is outside the protected environment, particular attributes for the client computing device; and 
 determining, based on the particular attributes, whether to serve a particular media item to the client computing device. 
 
     
     
       15. The system of  claim 11 , the storage media further comprising instructions which when executed by one or more of the processors cause the system to perform storing attribute values for a plurality of different client computing devices in an attribute database in the protected environment. 
     
     
       16. The system of  claim 11 , the first criterion being a minimum number of instances in the second data of a particular personal data record having the status value. 
     
     
       17. The system of  claim 11 , the second criterion being a maximum fraction of population at risk. 
     
     
       18. The system of  claim 17 , the storage media further comprising instructions which when executed by one or more of the processors cause the system to perform computing the maximum fraction of population at risk as a quotient of a number of instances in the subset of the first data of a patient having the status value and a number of positive predictions of the status value from applying the particular machine learning system to each of the plurality of personal data records in the first data. 
     
     
       19. The system of  claim 11 , the storage media further comprising instructions which when executed by one or more of the processors cause the system to perform:
 training the particular machine learning system with a first set of parameters; and 
 determining that the particular machine learning system does not satisfy the second criterion and, in response, training the particular machine learning system using a second set of parameters. 
 
     
     
       20. The system of  claim 11 , the status being one of a particular medical diagnosis or a particular prescription.