Systems and methods for self-supervised learning based on naturally-occurring patterns of missing data

Abstract

Disclosed is a method comprising accessing, by a machine learning system, a set of data records for a plurality of users, the data records representative of physical statistics measured for each of the plurality of users over a time period. At least a subset of the data records comprises patterns of missing data for at least a portion of the time period. The method also comprises generating a set of masked data records by masking a subset of the data records in accordance with a pattern of natural missingness from a data record. The method also comprises generating, by the machine learning system, a set of learned representations from at least the set of masked data records. Finally, the method comprises fine tuning, by the machine learning system, a machine learning model using the set of learned representations, the machine learning model configured to perform a downstream machine learning task.

What is claimed is: 
     
       1. A method comprising:
 (a) identifying, based at least in part on one or more demographics of a target user, a population from a larger group of users, wherein the population comprises one or more digital twins of the target user, and wherein the population is characterized by having a common demographic; 
 (b) accessing, by a machine learning system, a set of data records for a plurality of users of the population, the set of data records representative of physical statistics measured for each user of the plurality of users of the population over a time period, wherein the physical statistics for each user of the plurality of users of the population are measured using a wearable device associated with each user of the plurality of users of the population; 
 (c) generating a set of masked data records by masking at least a subset of the set of data records in accordance with a pattern of missingness from the set of data records, wherein the pattern of missingness from the set of data records corresponds to periods of disuse or deactivation of the wearable device associated with each user of the plurality of users of the population; 
 (d) generating, by the machine learning system, a plurality of learned representations from at least the set of masked data records; and 
 (e) fine tuning, by the machine learning system, a machine learning model using the plurality of learned representations, the machine learning model configured to perform a downstream machine learning task that comprises imputing missing data from a wearable device associated with the target user, thereby generating complete data for the target user. 
 
     
     
       2. The method of  claim 1 , wherein a data record of the subset of the set of data records comprises missing data different from the pattern of missingness. 
     
     
       3. The method of  claim 2 , further comprising using a learned representation of the plurality of learned representations to identify another subset of the set of data records using one or more clustering or segmentation techniques to one or more of: (i) perform event detection, (ii) detect or predict onset of an acute health condition, (iii) monitor a chronic health condition, (iv) detect trends, (v) detect outliers, or (vi) identify users that closely resemble one another in terms of health, behavior, or activity. 
     
     
       4. The method of  claim 1 , wherein generating the set of masked data records comprises determining a level of similarity between a data record of the set of data records and a data record of the subset of the set of data records. 
     
     
       5. The method of  claim 1 , wherein generating the set of masked data records comprises dividing the subset of the set of data records into a plurality of groups using one or more segmentation or clustering techniques, wherein missingness of each data record of the subset of the set of data records is used to mask another data record of the subset of the set of data records that is within a common group of the plurality of groups when generating a training dataset. 
     
     
       6. The method of  claim 1 , wherein the physical statistics comprise physiological data, wherein the physiological data comprise one or more of: resting heart rate, current heart rate, heart rate variability, respiration rate, galvanic skin response, skin temperature, or blood oxygen level. 
     
     
       7. The method of  claim 1 , wherein the physical statistics comprise behavioral data, and wherein the behavioral data comprise one or more of: daily number of steps, distance walked, time active, exercise amount, exercise type, time slept, number of times sleep was interrupted, sleep start times, sleep end times, napping, or resting. 
     
     
       8. The method of  claim 1 , wherein the set of data records comprise time series data. 
     
     
       9. The method of  claim 1 , wherein the wearable device associated with each user of the plurality of users of the population comprises a personal health sensor device. 
     
     
       10. A system comprising a computing device comprising at least one processor and instructions executable by the at least one processor to cause the at least one processor to perform operations comprising:
 (a) identifying, based at least in part on one or more demographics of a target user, a population from a larger group of users, wherein the population comprises one or more digital twins of the target user, and wherein the population is characterized by having a common demographic; 
 (b) accessing, by a machine learning system, a set of data records for a plurality of users of the population, the set of data records representative of physical statistics measured for each user of the plurality of users of the population over a time period, wherein the physical statistics for each user of the plurality of users of the population are measured using a wearable device associated with each user of the plurality of users of the population; 
 (c) for each data record of a subset of the set of data records:
 (i) identifying, by the machine learning system, a portion of the time period associated with a pattern of missing data from the set of data records, wherein the pattern of missing data corresponds to periods of disuse or deactivation of the wearable device associated with each user of the plurality of users of the population, and 
 (ii) generating, by the machine learning system, a masked data record by masking a portion of an additional data record of the set of data records corresponding to the identified portion of the time period, wherein the masking of the additional data record of the subset of the set of data records causes the additional data record to resemble the pattern of missing data; 
 
 (d) generating, by the machine learning system, a training dataset comprising both of at least the portion of the additional data record and the corresponding generated masked data record for each data record of the subset of the set of data records; 
 (e) training, by the machine learning system, a machine learning model using the generated training dataset, the machine learning model configured to predict, for a received data record comprising masked data, imputed data corresponding to data of the received data record obscured based on the masked data; 
 (e) generating, by the machine learning model, a plurality of learned representations, wherein the plurality of learned representations are associated with the prediction of the imputed data; and 
 (g) fine-tuning, by the machine learning system, a learned representation of the plurality of learned representations to a downstream machine learning task, wherein the downstream machine learning task comprises processing a set of data records from a wearable device associated with the target user, thereby generating complete data for the target user. 
 
     
     
       11. The system of  claim 10 , wherein the additional data record comprises missing data different from the identified portion of the time period of the corresponding data record. 
     
     
       12. The system of  claim 10 , wherein the instructions are executable by the at least one processor to cause the at least one processor to perform operations further comprising: determining a level of similarity between a current data record and the additional data record. 
     
     
       13. The system of  claim 10 , wherein the instructions are executable by the at least one processor to cause the at least one processor to perform operations further comprising: dividing the subset of the set of data records into a plurality of groups using one or more segmentation or clustering techniques, wherein the pattern of missing data from the set of data records is used to mask other data records within a common group of the plurality of groups when generating the training dataset. 
     
     
       14. The system of  claim 10 , wherein the missing data is a result of missingness arising from user behavioral patterns. 
     
     
       15. The system of  claim 10 , wherein the physical statistics comprise physiological data, wherein the physiological data comprise one or more of: resting heart rate, current heart rate, heart rate variability, respiration rate, galvanic skin response, skin temperature, or blood oxygen level. 
     
     
       16. The system of  claim 10 , wherein the physical statistics comprise behavioral data, wherein the behavioral data comprise one or more of daily number of steps, distance walked, time active, exercise amount, exercise type, time slept, number of times sleep was interrupted, sleep start times, sleep end times, napping, or resting. 
     
     
       17. The system of  claim 10 , wherein the set of data records comprise time series data. 
     
     
       18. The system of  claim 10 , wherein the set of data records are generated by personal health sensor devices. 
     
     
       19. The system of  claim 18 , wherein the wearable device associated with each user of the plurality of users of the population comprises a personal health sensor device. 
     
     
       20. The system of  claim 10 , wherein the instructions are executable by the at least one processor to cause the at least one processor to perform operations further comprising: generating multiple training datasets over a plurality of iterations. 
     
     
       21. A non-transitory computer-readable storage media encoded with instructions executable by one or more processors to cause the at least one processor to perform operations comprising:
 (a) identifying, based at least in part on one or more demographics of a target user, a population from a larger group of users, wherein the population comprises one or more digital twins of the target user, and wherein the population is characterized by having a common demographic; 
 (b) accessing, by a machine learning system, a set of data records for a plurality of users of the population, the data records representative of physical statistics measured for each user of the plurality of users of the population over a time period, wherein the physical statistics for each user of the plurality of users of the population are measured using a wearable device associated with each user of the plurality of users of the population; 
 (c) for each data record of a subset of data records:
 (i) identifying, by the machine learning system, a portion of the time period corresponding to missing data, and 
 (ii) generating, by the machine learning system, a masked data record by masking a portion of an additional data record of the set of data records corresponding to the identified portion of the time period, to resemble a pattern of missing data from the set of data records, wherein the pattern of missing data corresponds to periods of disuse or deactivation of the wearable device associated with each user of the plurality of users of the population; 
 
 (d) generating, by the machine learning system, a training dataset comprising both of at least the portion of the additional data record and the corresponding generated masked data record for each data record of the subset of the set of data records; 
 (e) training, by the machine learning system, a machine learning model using the generated training dataset, the machine learning model configured to predict, for a received data record comprising masked data, imputed data corresponding to data of the received data record obscured based on the masked data; 
 (f) generating, by the machine learning system, a plurality of learned representations, wherein the plurality of learned representations are associated with the prediction of the imputed data as a result of the imputation of the masked data in the pattern of missing data; and 
 (g) fine-tuning, by the machine learning machine learning system, a learned representation of the plurality of learned representations to a downstream machine learning task, wherein the downstream machine learning task comprises processing a set of data records from a wearable device associated with the target user, thereby generating complete data for the target user. 
 
     
     
       22. A computer-implemented method of training a machine learning model to generate inferences from wearable sensor data, comprising:
 (a) identifying, based at least in part on one or more demographics of a target user, a population from a larger group of users, wherein the population comprises one or more digital twins of the target user, and wherein the population is characterized by having a common demographic; 
 (b) retrieving a first set of wearable sensor data for each subject of a plurality of subjects of the population, wherein the first set of wearable sensor data for each subject of the plurality of subjects of the population is measured using a wearable device associated with each subject of the plurality of subjects of the population; 
 (c) selectively masking portions of at least a subset of the first set of wearable sensor data, wherein the masked portions of at least the subset of the first set of wearable sensor data are associated with periods of missing data and wherein the masking of the portions of at least the subset of the first set of the wearable sensor data causes the portions of at least the subset of the first set of the wearable sensor data to resemble a pattern of missing data from the set of data records, wherein the pattern of missing data corresponds to periods of disuse or deactivation of the wearable device associated with each user of the plurality of users of the population; 
 (d) creating a training set comprising at least the subset of the first set of wearable sensor data; 
 (e) training the machine learning model to impute data to the masked portions of at least the subset of the first set of wearable sensor data, thereby producing at least one learned representation from the training; and 
 (f) fine-tuning the at least one learned representation by using the machine learning model to a downstream machine learning task, wherein the downstream machine learning task comprises processing a second set of wearable sensor data from a wearable device associated with the target user, thereby generating complete data for the target user. 
 
     
     
       23. The method of  claim 22 , wherein the downstream machine learning task is imputation, regression, segmentation, or classification. 
     
     
       24. The method of  claim 22 , wherein the machine learning model comprises an attention mechanism. 
     
     
       25. The method of  claim 24 , wherein the attention mechanism is a multi-head attention mechanism. 
     
     
       26. The method of  claim 22 , wherein at least a portion of the first wearable sensor data is synthetically generated, wherein synthetically generating the at least the portion of the first wearable sensor data comprises:
 (i) providing a set of time series wearable sensor data; 
 (ii) generating a plurality of embeddings from the time series wearable data, wherein an embedding of the plurality of embeddings comprises a sequence of values, wherein a value of the sequence of values is associated with a position of a set of positions; and 
 (iii) predicting a value for a position of the set of positions not associated with a value of the sequence of values by processing the plurality of embeddings with a second machine learning model, wherein the second machine learning model comprises an attention mechanism, wherein at least a portion of an attention weight matrix generated from processing the plurality of embeddings is masked. 
 
     
     
       27. The method of  claim 26 , wherein the set of positions comprises a set of positions in time, and wherein a position in time of the set of positions in time corresponds to a future position in time. 
     
     
       28. The method of  claim 26 , wherein the set of positions comprises a set of positions in time, and wherein a position in time of the set of positions in time corresponds to a masked position in time. 
     
     
       29. The method of  claim 1 , wherein the common demographic is a common health condition. 
     
     
       30. The method of  claim 1 , wherein the downstream machine learning task further comprises: predicting onset of a health condition for the target user based at least in part on the complete data for the target user.