Analytics framework for selection and execution of analytics in a distributed environment

Abstract

Mechanisms are provided to implement a multi-layer analytics framework. The multi-layer analytics framework obtains a plurality of analytics from one or more analytics source computing systems, at least two analytics being written in different computer programming languages. The multi-layer analytics framework applies a wrapper to each of the analytics in the plurality of analytics to thereby generate wrapped analytics. The wrapper provides a unified interface for executing the analytics in the plurality of analytics regardless of the particular computer programming language used to create the analytics. The multi-layer analytics framework registers the wrapped analytics in an analytics registry, and executes an analytics pipeline comprising wrapped analytics in the analytics registry to perform an analytics operation based on the unified interface of the wrappers of the wrapped analytics.

What is claimed is: 
     
       1. A method, in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions executed by the at least one processor to cause the at least one processor to implement a multi-layer analytics framework, the method comprising:
 receiving a request to perform an analytics operation on input data, the request specifying features of wrapped analytics to be used in generating an analytics pipeline; 
 automatically composing the analytics pipeline at least by selecting wrapped analytics from a wrapped analytics registry based on the specified features, wherein composing the analytics pipeline comprises correlating wrapped analytics based on analytics data models associated with each of the selected wrapped analytics, wherein the analytics data models specify metadata about the corresponding wrapped analytic; and 
 automatically executing the composed analytics pipeline on the input data to generate results data of the analytics operation. 
 
     
     
       2. The method of  claim 1 , further comprising applying, by the multi-layer analytics framework, a wrapper to each analytic in a plurality of analytics to thereby generate the wrapped analytics, and wherein wrappers associated with the at least two analytics are written in a same computer programming language as the analytic which is wrapped by the wrapper and implement a domain standardization model to provide the unified interface. 
     
     
       3. The method of  claim 2 , wherein the wrapper comprises a plurality of application programming interfaces (APIs) for various programming languages, and wherein the APIs provide a unified form to represent input parameters and configuration for analytics and a unified form to write analytics results and explanation. 
     
     
       4. The method of  claim 1 , wherein composing the analytics pipeline based on the specified features further comprises correlating inputs and outputs of the selected wrapped analytics such that a required input to a first selected wrapped analytic of the analytics pipeline is provided by at least one of the input dataset or an output of another second selected wrapped analytic. 
     
     
       5. The method of  claim 1 , wherein the analytics pipeline comprises a multi-modal analytics pipeline comprising wrapped analytics of different modalities, wherein a set of the wrapped analytics execute in parallel to other wrapped analytics of the analytics pipeline, and wherein results of the parallel execution of the set of wrapped analytics of the analytics pipeline are combined in append mode logic. 
     
     
       6. The method of  claim 5 , wherein the different modalities comprise at least two of textual analytics, imaging analytics, or reasoning analytics. 
     
     
       7. The method of  claim 1 , wherein the analytics data model corresponding to a wrapped analytic specifies at least one of metadata defining analytics pipelines with which the corresponding wrapped analytic is usable, knowledge addressed by the corresponding wrapped analytic, development and integration details of the corresponding wrapped analytic, environment setup details of the corresponding wrapped analytic, analytics delivery and execution models associated with the corresponding wrapped analytic, and performance details of the wrapped analytic. 
     
     
       8. The method of  claim 1 , wherein the analytics pipeline is executed in a distributed manner by a plurality of computing devices in accordance with a directed acyclic graph (DAG) representation of the analytics pipeline indicating which portions of the analytics pipeline are executed in parallel on different computing devices and which portions of the analytics pipeline are executed serially. 
     
     
       9. The method of  claim 8 , further comprising:
 analyzing the DAG representation of the analytics pipeline to identify performance efficiency characteristics of the selected wrapped analytics and the analytics pipeline; and 
 distributing the selected wrapped analytics to a plurality of computing devices based on the identified performance efficiency characteristics. 
 
     
     
       10. The method of  claim 1 , further comprising:
 applying, by the multi-layer analytics framework, a wrapper to each analytic in a plurality of analytics to thereby generate the wrapped analytics; and 
 registering the wrapped analytics in a wrapped analytics repository at least by, for each wrapped analytic, generating an index entry in an index data structure, the index entry specifying, for a corresponding wrapped analytic, inputs to the wrapped analytic, outputs of the wrapped analytic, and one or more performance characteristics of the wrapped analytic, in a searchable manner, and wherein automatically composing the analytics pipeline comprises searching index entries in the index data structure for wrapped analytics index entries having characteristics that match one or more of the specified features of wrapped analytics specified in the request. 
 
     
     
       11. The method of  claim 1 , wherein composing the analytics pipeline comprises:
 searching a pipeline registry, having entries specifying characteristics of previously generated analytics pipelines, based on the specified features of wrapped analytics specified in the request, to determine if a matching entry in the pipeline registry having characteristics of a previously generated analytics pipeline that match the specified features is present; 
 in response to finding a matching entry in the pipeline registry, retrieving a corresponding analytics pipeline, and a corresponding directed acyclic graph data structure associated with the matching entry, as the composed analytics pipeline; and 
 in response to not finding a matching entry in the pipeline registry, performing the automatic composing of the analytics pipeline based on the specified features and based on inputs and outputs of the selected wrapped analytics. 
 
     
     
       12. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement a multi-layer analytics framework, which operates to:
 receive a request to perform an analytics operation on input data, the request specifying features of wrapped analytics to be used in generating an analytics pipeline; 
 automatically compose the analytics pipeline at least by selecting wrapped analytics from a wrapped analytics registry based on the specified features, wherein composing the analytics pipeline comprises correlating wrapped analytics based on analytics data models associated with each of the selected wrapped analytics, wherein the analytics data models specify metadata about the corresponding wrapped analytic; and 
 automatically execute the composed analytics pipeline on the input data to generate results data of the analytics operation. 
 
     
     
       13. The computer program product of  claim 12 , wherein the computer readable program further causes the computing device to apply, by the multi-layer analytics framework, a wrapper to each analytic in a plurality of analytics to thereby generate the wrapped analytics, and wherein wrappers associated with the at least two analytics are written in a same computer programming language as the analytic which is wrapped by the wrapper and implement a domain standardization model to provide the unified interface. 
     
     
       14. The computer program product of  claim 13 , wherein the wrapper comprises a plurality of application programming interfaces (APIs) for various programming languages, and wherein the APIs provide a unified form to represent input parameters and configuration for analytics and a unified form to write analytics results and explanation. 
     
     
       15. The computer program product of  claim 12 , wherein composing the analytics pipeline based on the specified features further comprises correlating inputs and outputs of the selected wrapped analytics such that a required input to a first selected wrapped analytic of the analytics pipeline is provided by at least one of the input dataset or an output of another second selected wrapped analytic. 
     
     
       16. The computer program product of  claim 12 , wherein the analytics pipeline comprises a multi-modal analytics pipeline comprising wrapped analytics of different modalities, wherein a set of the wrapped analytics execute in parallel to other wrapped analytics of the analytics pipeline, and wherein results of the parallel execution of the set of wrapped analytics of the analytics pipeline are combined in append mode logic. 
     
     
       17. The computer program product of  claim 16 , wherein the different modalities comprise at least two of textual analytics, imaging analytics, or reasoning analytics. 
     
     
       18. The computer program product of  claim 12 , wherein the analytics data model corresponding to a wrapped analytic specifies at least one of metadata defining analytics pipelines with which the corresponding wrapped analytic is usable, knowledge addressed by the corresponding wrapped analytic, development and integration details of the corresponding wrapped analytic, environment setup details of the corresponding wrapped analytic, analytics delivery and execution models associated with the corresponding wrapped analytic, and performance details of the wrapped analytic. 
     
     
       19. The computer program product of  claim 12 , wherein the analytics pipeline is executed in a distributed manner by a plurality of computing devices in accordance with a directed acyclic graph (DAG) representation of the analytics pipeline indicating which portions of the analytics pipeline are executed in parallel on different computing devices and which portions of the analytics pipeline are executed serially. 
     
     
       20. An apparatus comprising:
 a processor; and 
 a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to implement a multi-layer analytics framework, which operates to: 
 receive a request to perform an analytics operation on input data, the request specifying features of wrapped analytics to be used in generating an analytics pipeline; 
 automatically compose the analytics pipeline at least by selecting wrapped analytics from a wrapped analytics registry based on the specified features, wherein composing the analytics pipeline comprises correlating wrapped analytics based on analytics data models associated with each of the selected wrapped analytics, wherein the analytics data models specify metadata about the corresponding wrapped analytic; and 
 automatically execute the composed analytics pipeline on the input data to generate results data of the analytics operation.