Interest-Driven Business Intelligence Systems and Methods of Data Analysis Using Interest-Driven Data Pipelines

Abstract

Interest-driven Business Intelligence (BI) systems in accordance with embodiments of the invention are illustrated. In one embodiment of the invention, a data processing system includes raw data storage containing raw data, metadata storage containing metadata that describes the raw data, and an interest-driven data pipeline that is automatically compiled to generate reporting data using the raw data, wherein the interest-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report specification defined using the metadata.

What is claimed is: 
     
         1 . A data processing system, comprising:
 raw data storage containing raw data;   metadata storage containing metadata that describes the raw data; and   an interest-driven data pipeline that is automatically compiled to generate reporting data using the raw data;   wherein the interest-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report specification defined using the metadata.   
     
     
         2 . The data processing system of  claim 1 , wherein the interest-driven data pipeline is automatically compiled based upon at least one change selected from the group consisting of a change to the reporting data requirements, a change to the metadata, and a change to the raw data. 
     
     
         3 . The data processing system of  claim 1 , further comprising:
 source data storage configured to store source data; and   wherein the interest-driven data pipeline is configured to create source data by performing an extract, transform, and load (ETL) process on raw data using a source model.   
     
     
         4 . The data processing system of  claim 4 , further comprising:
 aggregate data storage configured to store aggregate data; and   wherein the interest-driven data pipeline is configured to generate aggregate data by performing aggregations using the source data.   
     
     
         5 . The data processing system of  claim 4 , wherein the interest-driven data pipeline is configured to generate aggregations utilizing the reporting data requirements. 
     
     
         6 . The data processing system of  claim 1 , further comprising an intermediate processing layer, wherein the intermediate processing layer comprises:
 a data manager configured to store data models; and   an interest-driven data pipeline compiler.   
     
     
         7 . The data processing system of  claim 6 , wherein the intermediate processing layer accesses data within the raw data storage. 
     
     
         8 . The data processing system of  claim 6 , wherein the interest-driven data pipeline is automatically compiled utilizing the interest-driven data pipeline compiler. 
     
     
         9 . The data processing system of  claim 6 , wherein the intermediate processing layer is configured to perform raw data intake comprising updating raw data and updating reporting data. 
     
     
         10 . The data processing system of  claim 1 , wherein:
 the data processing system is configured as an interest-driven business intelligence system comprising:
 a business intelligence reporting engine; 
 wherein the business intelligence reporting engine includes reporting data storage configured to store reporting data. 
   
     
     
         11 . The interest-driven business intelligence system of  claim 10 , wherein the business intelligence reporting engine comprises a user interface configured to display data and receive requests for data. 
     
     
         12 . The interest-driven business intelligence system of  claim 10 , further comprising:
 report specification storage configured to store report specifications;   wherein the interest-driven business intelligence system is configured to receive at least one report specification.   
     
     
         13 . The interest-driven business intelligence system of  claim 12 , wherein the interest-driven data pipeline is configured to automatically generate reporting data requirements using at least one report specification. 
     
     
         14 . The interest-driven business intelligence system of  claim 13 , wherein the business intelligence reporting engine is configured to generate a report utilizing the report specification and the reporting data. 
     
     
         15 . The interest-driven business intelligence system of  claim 14  comprising an interest-driven data pipeline compiler, wherein the interest-driven data pipeline is configured to:
 convert the raw data into source data utilizing an extract, transform, and load (ETL) process; 
 utilize the reporting data requirements to generate aggregation processes that are applied to the source data to generate aggregate data; and 
 generate a data model based upon the reporting data requirements and to populate the data model using the aggregate data to generate reporting data; 
 wherein the interest-driven data pipeline compiler is configured to automatically compile the interest-driven data pipeline in response to the interest-driven business intelligence system receiving a report specification modifying the reporting data requirements. 
 
     
     
         16 . The interest-driven business intelligence system of  claim 15 , wherein the interest-driven data pipeline compiler is configured to update the ETL process in response to the modifications to the reporting data requirements. 
     
     
         17 . The interest-driven business intelligence system of  claim 15 , wherein the interest-driven data pipeline compiler is configured to generate new aggregations in response to the modifications to the reporting data requirements. 
     
     
         18 . The interest-driven business intelligence system of  claim 15 , wherein the interest-driven data pipeline compiler is configured to generate a new data model in response to the modifications to the reporting data requirements. 
     
     
         19 . The interest-driven business intelligence system of  claim 15 , wherein the report comprises at least one visualization, where the at least one visualization is configured to process and display reporting data. 
     
     
         20 . A method for creating a report utilizing an interest-driven business intelligence system, comprising:
 storing raw data in raw data storage;   receiving report definitions generated utilizing metadata describing the raw data using the interest-driven business intelligence system;   automatically generating reporting data requirements derived from report definitions using the interest-driven business intelligence system;   automatically compiling an interest-driven data pipeline that is part of the interest-driven business intelligence system using the reporting data requirements and the raw data;   generating reporting data from the reporting data requirements using the interest-driven data pipeline;   populating reports with the reporting data using the interest-driven business intelligence system;   providing interactive exploration of the reports using the interest-driven business intelligence system; and   automatically compiling the interest-driven data pipeline to generate additional reporting data in response to changes in the reporting data requirements defined during the interactive exploration of the reports using the interest-driven business intelligence system.