Interest-driven business intelligence systems and methods of data analysis using interest-driven data pipelines

Abstract

Interest-driven Business Intelligence (BI) systems in accordance with embodiments of the invention are illustrated. In one embodiment of the invention, a data processing system includes raw data storage containing raw data, metadata storage containing metadata that describes the raw data, and an interest-driven data pipeline that is automatically compiled to generate reporting data using the raw data, wherein the interest-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report specification defined using the metadata.

What is claimed is: 
     
       1. An interest-driven business intelligence system, comprising:
 a raw data storage system configured to contain raw data and perform extract, transform, and load (ETL) processes, where the raw data comprises unstructured data; 
 metadata storage configured to contain metadata that describes the raw data; 
 aggregate data storage configured to contain aggregate data, where aggregate data comprises structured data generated using ETL processes from the raw data; 
 an intermediate processing layer; 
 a business intelligence reporting engine; 
 wherein the intermediate processing layer is configured to automatically:
 generate metadata describing the raw data; 
 derive reporting data requirements from the at least one report specification based on the metadata; and 
 automatically compile an interest-driven data pipeline based upon the reporting data requirements, where automatically compiling the interest-driven data pipeline comprises:
 generating ETL processing jobs to generate aggregate data from the raw data by:
 filtering the raw data using the structure of the raw data; and 
 applying transformations to the raw data based on the structure of the raw data; 
 
 storing the aggregate data in the aggregate data storage; 
 generating reporting data including reporting data satisfying the reporting data requirements using the aggregate data; and 
 storing the reporting data in a data mart within the intermediate processing layer for exploration by the business intelligence reporting engine; 
 
 
 wherein the business intelligence reporting engine is configured to:
 receive metadata describing the raw data from the intermediate processing layer; and 
 generate a user interface enabling user exploration of the metadata to define at least one report specification, where the user exploration involves selection of additional reporting data using the metadata; and 
 
 wherein the intermediate processing layer is further configured to:
 automatically update the reporting data requirements based upon the additional reporting data selected using the metadata via the business intelligence reporting engine; 
 automatically recompile the interest-driven data pipeline in real time to generate the additional reporting data selected using the metadata in response to the changes in the updated reporting data requirements by:
 automatically generating ETL jobs to generate updated aggregate data from the raw data and providing the ETL jobs to the raw data storage; 
 automatically generating the additional reporting data using the updated aggregate data; and 
 storing the additional reporting data in the data mart within the intermediate processing layer that includes the previously generated reporting; and 
 
 
 wherein the business intelligence reporting engine is further configured to generate at least one report based upon the at least one report specification using the reporting data stored within the data mart within the intermediate processing layer. 
 
     
     
       2. The interest-driven business intelligence system of  claim 1 , wherein the raw data storage system is a data warehouse. 
     
     
       3. The interest-driven business intelligence system of  claim 2 , wherein the data warehouse is implemented utilizing a system selected from the group consisting of a distributed computing system, a database management system, and a NoSQL database. 
     
     
       4. The interest-driven business intelligence system of  claim 2 , wherein the data warehouse is a distributed computing system implemented utilizing Hadoop. 
     
     
       5. The interest-driven business intelligence system of  claim 2 , wherein the data warehouse is configured to store data generated utilizing the intermediate processing layer. 
     
     
       6. The interest-driven business intelligence system of  claim 2 , wherein the intermediate processing layer is configured to generate data warehouse requests. 
     
     
       7. The interest-driven business intelligence system of  claim 6 , wherein the data warehouse requests are requests selected from the group consisting of Hive queries and MapReduce operations. 
     
     
       8. The interest-driven business intelligence system of  claim 1 , wherein the intermediate processing layer is implemented utilizing a system selected from the group consisting of a distributed computing system, a database management system, and a NoSQL database system. 
     
     
       9. The interest-driven business intelligence system of  claim 1 , wherein the business intelligence reporting engine is configured to display an indication based upon the interactive exploration of the at least one report. 
     
     
       10. The interest-driven business intelligence system of  claim 9 , wherein the indication is an estimate of the time needed to update the reporting data requirements.