Interest-Driven Business Intelligence Systems and Methods of Data Analysis Using Interest-Driven Data Pipelines

Abstract

Interest-driven Business Intelligence (BI) systems in accordance with embodiments of the invention are illustrated. In one embodiment of the invention, a data processing system includes raw data storage containing raw data, metadata storage containing metadata that describes the raw data, and an interest-driven data pipeline that is automatically compiled to generate reporting data using the raw data, wherein the interest-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report specification defined using the metadata.

What is claimed is: 
     
         1 . An interest-driven business intelligence system, comprising:
 raw data storage configured to contain raw data and perform extract, transform, and load (ETL) processes, where the raw data comprises unstructured data;   metadata storage configured to contain metadata that describes the raw data;   aggregate data storage configured to contain aggregate data, where aggregate data comprises structured data generated using ETL processes from the raw data;   an intermediate processing layer;   a business intelligence reporting engine;   wherein the intermediate processing layer is configured to automatically:
 generate metadata describing the raw data; 
 derive reporting data requirements from the at least one report specification based on the metadata; and 
 compile an interest-driven data pipeline based upon the reporting data requirements, where compiling the interest-driven data pipeline comprises:
 generating ETL processing jobs to generate aggregate data from the raw data by:
 filtering the raw data using the structure of the raw data; and 
 applying transformations to the raw data based on the structure of the raw data; 
 storing the aggregate data in the aggregate data storage; 
 
 generating reporting data including reporting data satisfying the reporting data requirements using the aggregate data; and 
 storing the reporting data in a data mart within the intermediate processing layer for exploration by the business intelligence reporting engine; 
 
   wherein the business intelligence reporting engine is configured to:
 receive metadata describing the raw data from the intermediate processing layer; and 
 generate a user interface enabling user exploration of the metadata to define at least one report specification, where the user exploration involves selection of additional reporting data using the metadata. 
   
     
     
         2 . The interest-driven business intelligence system of  claim 1 , wherein the raw data storage is a data warehouse. 
     
     
         3 . The interest-driven business intelligence system of  claim 2 , wherein the data warehouse is implemented utilizing a system selected from the group consisting of a distributed computing system, a database management system, and a NoSQL database. 
     
     
         4 . The interest-driven business intelligence system of  claim 2 , wherein the data warehouse is a distributed computing system implemented utilizing Hadoop. 
     
     
         5 . The interest-driven business intelligence system of  claim 2 , wherein the data warehouse is configured to store data generated utilizing the intermediate processing layer. 
     
     
         6 . The interest-driven business intelligence system of  claim 2 , wherein the intermediate processing layer is configured to generate data warehouse requests. 
     
     
         7 . The interest-driven business intelligence system of  claim 6 , wherein the data warehouse requests are requests selected from the group consisting of Hive queries and MapReduce operations. 
     
     
         8 . The interest-driven business intelligence system of  claim 1 , wherein the intermediate processing layer is implemented utilizing a system selected from the group consisting of a distributed computing system, a database management system, and a NoSQL database system. 
     
     
         9 . The interest-driven business intelligence system of  claim 1 , wherein the business intelligence reporting engine is configured to display an indication based upon the interactive exploration of the at least one report. 
     
     
         10 . The interest-driven business intelligence system of  claim 9 , wherein the indication is an estimate of the time needed to update the reporting data requirements. 
     
     
         11 . A method for creating a report utilizing an interest-driven business intelligence system, comprising:
 storing raw data in raw data storage using an interest-driven business intelligence system, where the raw data comprises unstructured data;   generating metadata describing the raw data using the interest-driven business intelligence system;   receiving report definitions generated utilizing the metadata describing the raw data using the interest-driven business intelligence system;   automatically generating reporting data requirements derived from report definitions using the interest-driven business intelligence system;   automatically compiling an interest-driven data pipeline that is part of the interest-driven business intelligence system using the reporting data requirements and the raw data;   generating reporting data from the reporting data requirements using the interest-driven data pipeline, where the reporting data comprises structured data generated using the raw data; and   storing the generated reporting data in a data mart within the interest-driven business intelligence system.