Systems and methods for imputing missing values in data sets

Abstract

A computer readable medium includes a data set with data stored in rows and N columns. Each of the rows is associated with one individual patient. Each of the N columns is associated with one type of data for patients. One or more processors is configured to: initialize missing values in M ones of the N columns with M values for the M ones of the N columns, respectively; generate M mathematical models for the M ones of the N columns having one or more missing values; for each of the rows having one or more missing values, update ones of the M values for the M ones of the N columns; and fill missing values in the M ones of the N columns with the M values, respectively.

What is claimed is: 
     
         1 . A system comprising:
 a computer readable medium including a data set with data stored in rows and columns,
 wherein each of the rows is associated with one individual patient, and 
 wherein each of the columns is associated with one type of data for patients; and 
   one or more processors configured to:
 (i) initialize missing values in ones of the columns in the data; 
 (ii) generate mathematical models for ones of the columns having one or more missing values, respectively, based on non-missing values of other ones of the columns in the data set; 
 (iii) for each of the rows having one or more missing values in one or more of the columns, respectively, update ones of the values for the one or more of the columns based on (a) non-missing values of that row, (b) the one or more of the mathematical models for the one or more of the columns, respectively, and (c) ones of the values for the other ones of the ones of the N columns with missing values; 
 (iv) updating the one or more mathematical models for the columns based on values in other columns and the updated ones of the values; and 
 (v) when an increase in performance of the mathematical models is less than a predetermined value, fill missing values in the M ones of the N columns in the data set with the M values, respectively. 
   
     
     
         2 . The system of  claim 1  wherein at least one of the columns includes categorical data that is limited to being in a first state, in a second state, or missing. 
     
     
         3 . The system of  claim 1  wherein at least one of the columns includes continuous data that is within a range of values or missing. 
     
     
         4 . The system of  claim 1  wherein the one or more processors are further configured to, when the increase in performance is greater than the predetermined value, repeat (iii) and (iv). 
     
     
         5 . The system of  claim 1  wherein the one or more processors are configured to determine the performance of the mathematical models after (iv). 
     
     
         6 . The system of  claim 1  wherein the one or more processors are configured to determine the performance using a root mean square error (RMSE) function. 
     
     
         7 . The system of  claim 1  wherein:
 the computer readable medium further includes a second data set with reserved data stored in rows and columns; and 
 the one or more processors are configured to determine the performance based on the second data set. 
 
     
     
         8 . The system of  claim 1  wherein at least one of the columns includes data regarding ICD-10 (International Statistical Classification of Diseases and Related Health Problems, 10th revision) codes. 
     
     
         9 . The system of  claim 1  wherein at least one of the columns includes indicators of results of laboratory tests. 
     
     
         10 . A system comprising:
 a computer readable medium including a data set with data stored in rows and columns,
 wherein each of the rows is associated with one individual patient, and 
 wherein each of the columns is associated with one type of data for patients; and 
   one or more processors configured to:
 (i) determine initial values for missing values, respectively, of a row; 
 (ii) generate a mathematical model for each column based on the data in one or more of the other columns; 
 (iii) for a column with a missing value in the row, predict the missing value for the column using the mathematical model of that column and values of the other columns in the row; 
 (iv) replace the missing value in the row with the prediction of the missing value; 
 (v) update the mathematical models after (iv); 
 (vi) determine an error metric based on the values of the row including the prediction of the missing value; and 
 (vii) repeat (iii)-(vi) until the error metric converges; and 
 (viii) after the error metric converges, fill missing values in the dataset using the mathematical models. 
   
     
     
         11 . The system of  claim 10  wherein the one or more processors are configured to determine the initial values randomly. 
     
     
         12 . The system of  claim 11  wherein the one or more processors are configured to determine the initial values further based on a frequency distribution of non-missing values of the columns. 
     
     
         13 . The system of  claim 11  wherein the one or more processors are configured to determine the initial values further based on a mean and a variance of the missing values of a column. 
     
     
         14 . The system of  claim 10  wherein at least one of the columns includes data regarding ICD-10 (International Statistical Classification of Diseases and Related Health Problems, 10th revision) codes. 
     
     
         15 . The system of  claim 10  wherein at least one of the columns includes indicators of results of laboratory tests. 
     
     
         16 . A method comprising:
 obtaining, from a computer readable medium, a data set with data stored in rows and columns,
 wherein each of the rows is associated with one individual patient, and 
 wherein each of the columns is associated with one type of data for patients; and 
   by one or more processors:
 (i) determining initial values for missing values, respectively, of a row; 
 (ii) generating a mathematical model for each column based on the data in one or more of the other columns; 
 (iii) for a column with a missing value in the row, predicting the missing value for the column using the mathematical model of that column and values of the other columns in the row; 
 (iv) replacing the missing value in the row with the prediction of the missing value; 
 (v) updating the mathematical models after (iv); 
 (vi) determining an error metric based on the values of the row including the prediction of the missing value; and 
 (vii) repeating (iii)-(vi) until the error metric converges; and 
 (viii) after the error metric converges, filling missing values in the dataset using the mathematical models. 
   
     
     
         17 . The method of  claim 16  wherein determining the initial values includes setting the initial values randomly. 
     
     
         18 . The method of  claim 17  wherein the one or more processors are configured to determine the initial values further based on a frequency distribution of non-missing values of the columns. 
     
     
         19 . The method of  claim 17  wherein the one or more processors are configured to determine the initial values further based on a mean and a variance of the missing values of a column. 
     
     
         20 . The method of  claim 16  wherein:
 a first one of the columns includes data regarding ICD-10 (International Statistical Classification of Diseases and Related Health Problems, 10th revision) codes; and 
 a second one of the columns includes indicators of results of laboratory tests.