Systems and methods for providing performance metadata in interest-driven business intelligence systems

Abstract

Interest-driven business intelligence server systems that provide performance metadata are described. In the disclosed embodiments, an interest-driven business intelligence server system receives a report specification. The report specification includes at least reporting data requirement. The interest-driven business intelligence server determines performance metadata information for an interest-driven data pipeline that is utilized to generate reporting data based on the report specification. The performance metadata information for the interest-driven data pipeline is transmitted to an interest-driven user visualization system by the interest-driven business intelligence server system.

What is claimed is: 
     
       1. An interest-driven business intelligence server system, comprising:
 a processor; 
 a memory connected to the processor and configured to store an interest-driven business intelligence application; 
 raw data storage configured to store raw data, wherein a portion of the raw data comprises unstructured data; 
 metadata storage configured to store data description metadata describing the raw data; and 
 report storage configured to store previously generated reports comprising previous reporting data and previous reporting data requirements; 
 wherein the interest-driven business intelligence application configures the processor to:
 receive a report specification, wherein the report specification comprises at least one reporting data requirement; 
 generate a sample of source data satisfying the at least one reporting data requirement from a sample of the raw data; 
 determine estimates of performance metadata information for an interest-driven data pipeline that is utilized to generate reporting data based at least in part on the report specification and the data description metadata describing the raw data, wherein the estimates of performance metadata information comprise an estimated data size of reporting data, wherein the determining of the performance metadata information satisfies the at least one reporting data requirement, and wherein the determining of the estimates of performance metadata information comprises to:
 apply required filters to the sample of the raw data; 
 apply required aggregations to the sample of the raw data; 
 determine performance metadata for the sample of source data, wherein the performance metadata includes performance metadata for the required aggregations and required filters and the required aggregations; and 
 determine the estimates of performance metadata for the sample of reporting data satisfying the at least one reporting data requirement based on the determined performance metadata of the sample of source data; 
 
 wherein the performance metadata includes performance metadata for the required aggregations and required filters and the required aggregations 
 provide the estimates of the performance metadata information for the interest-driven data pipeline to an interest-driven user visualization system, wherein the interest-driven user visualization system displays the estimated data size of the reporting data; 
 generate the reporting data utilizing the interest-driven data pipeline; and 
 store the reporting data and the at least one reporting data requirement in the report storage. 
 
 
     
     
       2. The system of  claim 1 , wherein the estimates of the performance metadata information for the interest-driven data pipeline provided to the interest-driven user visualization system includes performance metadata for at least one of the raw data, the reporting data, and a source data of the interest-driven data pipeline. 
     
     
       3. The system of  claim 1 , wherein the estimates of the performance metadata information for the interest-driven data pipeline provided to the interest-driven user visualization system includes historical performance metadata for the interest-driven data pipeline. 
     
     
       4. The system of  claim 3 , wherein the interest-driven business intelligence application configures the processor to retrieve the historical performance metadata from one of metadata accessible by the interest-driven business intelligence server system and a memory storing the historical performance metadata for the interest-driven data pipeline. 
     
     
       5. The system of  claim 1 , wherein:
 the interest-driven business intelligence application obtains the sample of the raw data by configuring the processor to:
 generate a mini job that is an ETL processing job that extracts a sufficient amount of data to provide an accurate sample of the raw data available; 
 provide the mini job to a distributed storage system; and 
 receive the sample of the raw data from the distributed storage system in response to the mini job; and 
 
 wherein the performance metadata for the sample of the raw data includes performance metadata for the mini job and the estimates of the performance metadata includes estimates for an ETL processing job for obtaining the raw data to satisfy the at least one reporting data requirement. 
 
     
     
       6. The system of  claim 5 , wherein:
 the interest-driven business intelligence application determines estimates of performance metadata for the raw data satisfying the at least one reporting data requirement by configuring the processor to retrieve historical performance metadata for the raw data satisfying the at least one reporting data requirement; and 
 the historical performance metadata is used to determine the estimates of the performance metadata for the raw data satisfying the at least one reporting data requirement. 
 
     
     
       7. The system of  claim 1 , wherein:
 the interest-driven business intelligence application determines the estimates of performance metadata for the source data satisfying the at least one reporting data requirement by configuring the processor to retrieve historical performance metadata for source data satisfying the at least one reporting data requirement; and 
 wherein the historical performance metadata is used to determine the estimates of the performance metadata for the source data satisfying the at least one reporting data requirement. 
 
     
     
       8. The system of  claim 1 , wherein the interest driven data sharing application configures the processor to:
 obtain the sample of source data satisfying the at least one reporting data requirement; 
 build a sample of reporting data satisfying the at least one reporting data requirement from the sample of source data; 
 determine performance metadata for the sample of reporting data; and 
 determine estimates of performance metadata for the sample of reporting data satisfying the at least one reporting data requirement based on the determined performance metadata of the sample of source data. 
 
     
     
       9. The system of  claim 8 , wherein the interest-driven business intelligence application determines estimates of performance metadata for the reporting data satisfying the at least one reporting data requirement by configuring the processor to:
 retrieve historical performance metadata for the reporting data satisfying the at least one reporting data requirement; and 
 wherein the historical performance metadata is used to determine the estimates of the performance metadata for the reporting data satisfying the at least one reporting data requirement. 
 
     
     
       10. The system of  claim 1 , wherein the interest-driven data pipeline includes transformation processes that transform a set of raw data into a set of reporting data and intermediate sets of data generated by the transformation processes. 
     
     
       11. The system of  claim 1 , wherein the interest-driven business intelligence application configures the processor to receive a modified report specification after the display of the size of the reporting data, wherein the modified report specification comprises a modification to the at least one reporting data requirement. 
     
     
       12. The system of  claim 11 , wherein the reporting data is generated based at least in part on the modified report specification. 
     
     
       13. A method performed by an interest-driven business intelligence server system to provide performance metadata for an interest-driven data pipeline comprising:
 receiving a report specification using an interest-driven business intelligence server system, wherein the report specification comprises at least one reporting data requirement; 
 generating a sample of source data satisfying the at least one reporting data requirement from a sample of raw data; 
 determining estimates of performance metadata information for an interest-driven data pipeline that is utilized to generate reporting data based at least in part on the report specification and data description metadata describing the raw data, wherein a metadata storage stores the data description metadata describing the raw data, wherein a raw data storage stores the raw data, wherein a portion of the raw data comprises unstructured data, and wherein the estimates of performance metadata information comprise an estimated data size of reporting data, wherein the determining of the performance metadata information satisfies the at least one reporting data requirement, and wherein the determining of the estimates of performance metadata information comprises:
 applying required filters to the sample of the raw data; 
 applying required aggregations to the sample of the raw data; 
 determining performance metadata for the sample of source data, wherein the performance metadata includes performance metadata for the required aggregations and required filters and the required aggregations; and 
 determining the estimates of performance metadata for the sample of reporting data satisfying the at least one reporting data requirement based on the determined performance metadata of the sample of source data; 
 
 providing the estimates of the performance metadata information for the interest-driven data pipeline to an interest-driven user visualization system, wherein the interest-driven user visualization system displays the estimated data size of the reporting data; 
 generating the reporting data utilizing the interest-driven data pipeline, and 
 storing the reporting data and the at least one reporting data requirement in a report storage wherein the report storage stores previously generated reports comprising previous reporting data and previous reporting data requirements. 
 
     
     
       14. The method of  claim 13 , further comprising:
 generating a mini job that is an ETL processing job that extracts a sufficient amount of data to provide an accurate sample of the raw data available using the interest-driven business intelligence server system; 
 providing the mini job to a distributed storage system using the interest-driven business intelligence server system; and 
 receiving the sample of raw data from the distributed storage system in response to the mini job using the interest-driven business intelligence server system; 
 wherein the performance metadata for the sample of raw data includes performance metadata for the mini job and the estimates of the performance metadata includes estimates for an ETL processing job for obtaining raw data to satisfy the at least one reporting data requirement.