Adaptive filtering and modeling via adaptive experimental designs to identify emerging data patterns from large volume, high dimensional, high velocity streaming data

Abstract

A system for identifying information in high dimensional, low latency streaming data having dynamically evolving data patterns. The system processes, continuously and in real-time, the streaming data. Processing includes filtering the data based on event data to identify diagnostic data points by comparing the event data with an experimental design matrix and performing a modeling operation using the identified diagnostic data points in order to identify efficiently any current and emerging patterns of relationships between at least one outcome variable and predictor variables. The at least one a-priori, pre-designed experimental design matrix is generated based on combinations of the predictor variables and at least one outcome variable. The experimental design matrix is also generated based on at least one of main effects, limitations, constraints, and interaction effects of the predictor variables and combinations.

What is claimed is: 
     
       1. A computer-implemented method for identifying information in high dimensional data streams having dynamically evolving data patterns, the method comprising:
 processing streaming data continuously, wherein the processing comprises:
 loading at least one a-priori, pre-designed experimental design matrix and at least one modeling operation into memory; 
 filtering the streaming data based on event data to identify diagnostic data points by comparing the event data with the at least one a-priori, pre-designed experimental design matrix; 
 performing the at least one modeling operation using the identified diagnostic data points to identify current and emerging patterns of relationships between at least one outcome variable and predictor variables; 
 
 wherein the at least one a-priori, pre-designed experimental design matrix is generated based on combinations of the predictor variables, wherein the combinations are based on the at least one outcome variable; and 
 wherein the at least one a-priori, pre-designed experimental design matrix is generated based on at least one of: 
 main effects of predictor variable values; 
 limitations of combinations of predictor variable values; 
 constraints of combination of predictor variable values; or 
 interaction effects between selected predictor variables. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein the at least one modeling operation is one of a prediction modeling operation or a clustering modeling operation. 
     
     
       3. The computer-implemented method of  claim 1 , wherein the limitations of the predictor variable values are determined based on range values for the predictor variables, wherein the predictor variable values are continuous predictor variables. 
     
     
       4. The computer-implemented method of  claim 1 , wherein the constraints of the combination of predictor variable values are based on a region of interest, wherein the predictor variable values are discrete predictor variable values. 
     
     
       5. The computer-implemented method of  claim 1 , wherein the at least one a-priori, pre-designed experimental design matrix is generated based on one of a space-filling design or an optimal experimental design. 
     
     
       6. The computer-implemented method of  claim 1 , wherein processing further comprises dynamically updating a visualization time window of the streaming data. 
     
     
       7. The computer-implemented method of  claim 1 , wherein the at least one a-priori, pre-designed experimental design matrix is generated based on main effects of predictor variable values. 
     
     
       8. The computer-implemented method of  claim 1 , wherein the at least one a-priori, pre-designed experimental design matrix is generated based on limitations of combinations of predictor variable values. 
     
     
       9. The computer-implemented method of  claim 1 , wherein the at least one a-priori, pre-designed experimental design matrix is generated based on constraints of combination of predictor variable values. 
     
     
       10. The computer-implemented method of  claim 1 , wherein the at least one a-priori, pre-designed experimental design matrix is generated based on interaction effects between selected predictor variables. 
     
     
       11. The computer-implemented method of  claim 1 , wherein the at least one modeling operation is a prediction modeling operation. 
     
     
       12. The computer-implemented method of  claim 1 , wherein the at least one modeling operation is a clustering modeling operation. 
     
     
       13. A system for identifying information in high dimensional data streams having dynamically evolving data patterns, the system comprising:
 one or more processors; 
 a memory coupled to the one or more processors and comprising instructions, which when performed by the one or more processors, cause the one or more processors to perform operations to:
 load at least one a-priori, pre-designed experimental design matrix and at least one modeling operation into memory; 
 filter streaming data based on event data continuously to identify diagnostic data points by comparing the event data with the at least one a-priori, pre-designed experimental design matrix; and 
 perform the at least one modeling operation using the identified diagnostic data points to identify current and emerging patterns of relationships between at least one outcome variable and predictor variables; 
 
 wherein the at least one a-priori, pre-designed experimental design matrix is generated based on combinations of the predictor variables, wherein the combinations are based on the at least one outcome variable; and 
 wherein the at least one a-priori, pre-designed experimental design matrix is selected based on at least one of:
 main effects of predictor variable values; 
 limitations of combinations of predictor variable values; 
 constraints of combination of predictor variable values; or 
 interaction effects between selected predictor variables. 
 
 
     
     
       14. The system of  claim 13 , wherein the at least one modeling operation is one of a prediction modeling operation or a clustering modeling operation. 
     
     
       15. The system of  claim 13 , wherein the limitations of the predictor variable values are determined based on range values for the predictor variables, wherein the predictor variable values are continuous predictor variables. 
     
     
       16. The system of  claim 13 , wherein the constraints of the combination of predictor variable values are based on a region of interest, wherein the predictor variable values are discrete predictor variable values. 
     
     
       17. The system of  claim 13 , wherein the at least one a-priori, pre-designed experimental design matrix is generated based on one of a space-filling design or an optimal experimental design. 
     
     
       18. The system of  claim 13 , wherein the instructions further cause the one or more processors to perform operations to dynamically update a visualization time window of the streaming data. 
     
     
       19. At least one non-transitory computer readable medium comprising instructions for identifying information in high dimensional streaming data having dynamically evolving data patterns, when executed by at least one processor, cause the at least one processor to perform operations to:
 load at least one a-priori, pre-designed experimental design matrix and at least one modeling operation into memory; 
 filter, continuously and real-time, streaming data based on event data to identify diagnostic data points by comparing the event data with the at least one a-priori, pre-designed experimental design matrix; and 
 perform, continuously and in real-time, the at least one modeling operation using the identified diagnostic data points to identify current and emerging patterns of relationships between at least one outcome variable and predictor variables; 
 wherein the at least one a-priori, pre-designed experimental design matrix is generated based on combinations of the predictor variables, wherein the combinations are based on the at least one outcome variable; and 
 wherein the at least one a-priori, pre-designed experimental design matrix is selected based on at least one of:
 main effects of predictor variable values; 
 limitations of combinations of predictor variable values; 
 constraints of combination of predictor variable values; or 
 interaction effects between selected predictor variables. 
 
 
     
     
       20. The at least one non-transitory computer readable medium of  claim 19 , wherein the at least one modeling operation is one of a prediction modeling operation or a clustering modeling operation.