System and method for analyzing big data activities

Abstract

A system and method for analyzing big data activities are disclosed. According to one embodiment, a system comprises a distributed file system for the entities and applications, wherein the applications include one or more of script applications, structured query language (SQL) applications, Not Only (NO) SQL applications, stream applications, search applications, and in-memory applications. The system further comprises a data processing platform that gathers, analyzes, and stores data relating to entities and applications. The data processing platform includes an application manager having one or more of a MapReduce Manage, a script applications manager, a structured query language (SQL) applications manager, a Not Only (NO) SQL applications manager, a stream applications manager, a search applications manager, and an in-memory applications manager. The application manager identifies if the applications are one or more of slow-running, failed, killed, unpredictable, and malfunctioning.

We claim: 
     
       1. A method, comprising:
 identifying using an application manager, resources used by an application for big data running on a cluster and providing MapReduce attempt success and failure totals for a workload; 
 identifying using a workflow manager, problems with the application by providing a status of how each workflow is performing against a service level agreement; 
 providing root-cause analysis and solutions for application problems and system-level problems; 
 providing alerts on the application, each workflow, users, tables, and queues; 
 tagging the application, each workflow, the tables, the users, and entities; 
 associating cluster activity with the application, each workflow, the tables, the users, the queues, machines, and the entities; and 
 simulating execution of the workload to identify a cheapest and a fastest infrastructure for the workload. 
 
     
     
       2. The method of  claim 1 , further comprising providing table names, amount of data, a number of records, and a number of tasks. 
     
     
       3. The method of  claim 1 , further comprising providing a MapReduce timeline, filters to display map, reduce, failed, killed, and successful tasks. 
     
     
       4. The method of  claim 1 , further comprising providing a skew view that shows a distribution of time and size of map and reduce tasks to identify a skew. 
     
     
       5. The method of  claim 1 , further comprising providing a view into each workflow to provide a comparison graph showing duration, resources used, data processed, and number of applications across various instances of each workflow. 
     
     
       6. The method of  claim 1 , further comprising providing a duration breakdown showing how time is spent in stages and phases of each workflow. 
     
     
       7. The method of  claim 1 , further comprising:
 displaying how queues are being used at any point in time; and 
 determining which queues are over-utilized and under-utilized. 
 
     
     
       8. The method of  claim 1 , further comprising determining cluster usage by users, data, resources, and queues. 
     
     
       9. The method of  claim 1 , further comprising providing a status of services running on a big data platform. 
     
     
       10. The method of  claim 1 , further comprising providing a list of applications that have caused inefficiencies such as resource wastage and inefficient join. 
     
     
       11. The method of  claim 1 , further comprising generating a top-N list to show which tables, columns, joins, and queries are used the most. 
     
     
       12. The method of  claim 1 , further comprising identifying tables and columns in the workload. 
     
     
       13. The method of  claim 1 , wherein the application is one of a script application, a structured query language (SQL) application, a Not Only (NO) SQL application, a stream application, a search application, and an in-memory application. 
     
     
       14. The method of  claim 1 , further comprising determining if the application is one or more of slow-running, failed, killed, unpredictable, and malfunctioning. 
     
     
       15. A non-transitory computer readable medium having stored thereon computer-readable instructions, and a processor coupled to the non-transitory computer readable medium, wherein the processor executes the instructions to:
 identify using an application manager, resources used by an application for big data running on a cluster and providing MapReduce attempt success and failure totals for a workload; 
 identify using a workflow manager, problems with the application by providing a status of how each workflow is performing against a service level agreement; 
 provide root-cause analysis and solutions for application problems and system-level problems; 
 provide alerts on the application, each workflow, users, tables, and queues; 
 tag the application, each workflow, the tables, the users, and entities; 
 associate cluster activity with the application, each workflow, the tables, the users, the queues, machines, and the entities; and 
 simulate execution of the workload to identify a cheapest and a fastest infrastructure for the workload. 
 
     
     
       16. The computer readable medium of  claim 15 , wherein the processor executes the instructions to provide table names, amount of data, a number of records, and a number of tasks. 
     
     
       17. The computer readable medium of  claim 15 , wherein the processor executes the instructions to provide a MapReduce timeline, filters to display map, reduce, failed, killed, and successful tasks. 
     
     
       18. The computer readable medium of  claim 15 , wherein the processor executes the instructions to provide a skew view that shows a distribution of time and size of map and reduce tasks to identify a skew. 
     
     
       19. The computer readable medium of  claim 15 , wherein the processor executes the instructions to provide a view into each workflow to provide a comparison graph showing duration, resources used, data processed, and number of applications across various instances of each workflow. 
     
     
       20. The computer readable medium of  claim 15 , wherein the processor executes the instructions to provide a duration breakdown showing how time is spent in stages and phases of each workflow. 
     
     
       21. The computer readable medium of  claim 15 , wherein the processor executes the instructions to:
 display how queues are being used at any point in time; and 
 determine which queues are over-utilized and under-utilized. 
 
     
     
       22. The computer readable medium of  claim 15 , wherein the processor executes the instructions to determine cluster usage by users, data, resources, and queues. 
     
     
       23. The computer readable medium of  claim 15 , wherein the processor executes the instructions to provide a status of services running on a big data platform. 
     
     
       24. The computer readable medium of  claim 15 , wherein the processor executes the instructions to provide a list of applications that have caused inefficiencies such as resource wastage and inefficient join. 
     
     
       25. The computer readable medium of  claim 15 , wherein the processor executes the instructions to generate a top-N list to show which tables, columns, joins, and queries are used the most. 
     
     
       26. The computer readable medium of  claim 15 , wherein the processor executes the instructions to identify tables and columns in the workload. 
     
     
       27. The computer readable medium of  claim 15 , wherein the application is one of a script application, a structured query language (SQL) application, a Not Only (NO) SQL application, a stream application, a search application, and an in-memory application. 
     
     
       28. The computer readable medium of  claim 15 , wherein the processor executes the instructions to determines if the application is one or more of slow-running, failed, killed, unpredictable, and malfunctioning.