Systems and methods of metadata monitoring and analysis

Abstract

A system and method of generating platform-dependent queries from a platform-agnostic query are disclosed. A data pipeline comprising a plurality of events is implemented. Each event in the plurality of events has a set of platform-dependent metadata associated therewith and each of the plurality of events is processed by one of a plurality of ingestion platforms. Metadata associated with each of the plurality of events is stored in a combined metadata repository. The combined metadata repository stores metadata extracted from two or more platforms in a first repository. A platform-agnostic query configured to obtain one or more metadata search results from the platform-dependent metadata is received and deployed to the first repository within the combined metadata repository. The platform-agnostic query is configured to return a result set including metadata obtained from each of the two or more platforms.

What is claimed is: 
     
       1. A system for generating item recommendations, comprising:
 a memory having instructions stored thereon, and a processor configured to read the instructions to:
 implement a data pipeline comprising a plurality of events, wherein each event in the plurality of events has a set of platform-dependent metadata associated therewith, and wherein each of the plurality of events is processed by one of a plurality of ingestion platforms; 
 store metadata associated with each of the plurality of events in a combined metadata repository, wherein the combined metadata repository stores metadata extracted from two or more platforms in a first repository; 
 receive a platform-agnostic query configured to obtain one or more metadata search results from the platform-dependent metadata; and 
 deploy the platform-agnostic query to the first repository within the combined metadata repository, wherein the platform-agnostic query is configured to return a result set including metadata obtained from each of the two or more platforms. 
 
 
     
     
       2. The system of  claim 1 , wherein the combined metadata repository comprises a schema repository. 
     
     
       3. The system of  claim 2 , wherein the processor is configured to read the instructions to:
 extract a first set of metadata from the data pipeline, wherein the first set of metadata is associated with a first platform; 
 generate a metadata schema for the first set of metadata; and 
 compare the generated metadata schema to a plurality of stored metadata schemas maintained by the combined metadata repository. 
 
     
     
       4. The system of  claim 1 , wherein the processor is configured to read the instructions to generate a first platform-dependent query from the platform-agnostic query, wherein the first platform-dependent query is configured to be deployed to the first repository. 
     
     
       5. The system of  claim 4 , wherein the first platform-dependent query is generated by applying a first set of conversion rules to the platform-agnostic query. 
     
     
       6. The system of  claim 1 , wherein the processor is configured to read the instructions to:
 receive one or more user query parameters; and 
 generate the platform-agnostic query in a predetermined platform-agnostic language. 
 
     
     
       7. The system of  claim 6 , wherein the platform-agnostic language is SQL. 
     
     
       8. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by a processor cause a device to perform operations comprising:
 implementing a data pipeline comprising a plurality of events, wherein each event in the plurality of events has a set of platform-dependent metadata associated therewith, and wherein each of the plurality of events is processed by one of a plurality of ingestion platforms; 
 storing metadata associated with each of the plurality of events in a combined metadata repository, wherein the combined metadata repository stores metadata extracted from two or more platforms in a first repository; 
 receiving a platform-agnostic query configured to obtain one or more metadata search results from the platform-dependent metadata; and 
 deploying the platform-agnostic query to the first repository within the combined metadata repository, wherein the platform-agnostic query is configured to return a result set including metadata obtained from each of the two or more platforms. 
 
     
     
       9. The non-transitory computer readable medium of  claim 8 , wherein the combined metadata repository comprises a schema repository. 
     
     
       10. The non-transitory computer readable medium of  claim 9 , wherein the instructions when executed by the processor cause the device to perform further operations comprising:
 extracting a first set of metadata from the data pipeline, wherein the first set of metadata is associated with a first platform; 
 generating a metadata schema for the first set of metadata; and 
 comparing the generated metadata schema to a plurality of stored metadata schemas maintained by the combined metadata repository. 
 
     
     
       11. The non-transitory computer readable medium of  claim 8 , wherein the instructions when executed by the processor cause the device to perform further operations comprising generating a first platform-dependent query from the platform-agnostic query, wherein the first platform-dependent query is configured to be deployed to the first repository. 
     
     
       12. The non-transitory computer readable medium of  claim 11 , wherein the first platform-dependent query is generated by applying a first set of conversion rules to the platform-agnostic query. 
     
     
       13. The non-transitory computer readable medium of  claim 8 , wherein the instructions when executed by the processor cause the device to perform further operations comprising:
 receiving one or more user query parameters; and 
 generating the platform-agnostic query in a predetermined platform-agnostic language. 
 
     
     
       14. The non-transitory computer readable medium of  claim 13 , wherein the platform-agnostic language is SQL. 
     
     
       15. A method, comprising:
 implementing a data pipeline comprising a plurality of events, wherein each event in the plurality of events has a set of platform-dependent metadata associated therewith, and wherein each of the plurality of events is processed by one of a plurality of ingestion platforms; 
 storing metadata associated with each of the plurality of events in a combined metadata repository, wherein the combined metadata repository stores metadata extracted from two or more platforms in a first repository; 
 receiving a platform-agnostic query configured to obtain one or more metadata search results from the platform-dependent metadata; and 
 deploying the platform-agnostic query to the first repository within the combined metadata repository, wherein the platform-agnostic query is configured to return a result set including metadata obtained from each of the two or more platforms. 
 
     
     
       16. The method of  claim 15 , wherein the combined metadata repository comprises a schema repository. 
     
     
       17. The method of  claim 16 , comprising:
 extracting a first set of metadata from the data pipeline, wherein the first set of metadata is associated with a first platform; 
 generating a metadata schema for the first set of metadata; and 
 comparing the generated metadata schema to a plurality of stored metadata schemas maintained by the combined metadata repository. 
 
     
     
       18. The method of  claim 15 , comprising generating a first platform-dependent query from the platform-agnostic query, wherein the first platform-dependent query is configured to be deployed to the first repository. 
     
     
       19. The method of  claim 18 , wherein the first platform-dependent query is generated by applying a first set of conversion rules to the platform-agnostic query. 
     
     
       20. The method of  claim 15 , comprising:
 receiving one or more user query parameters; and 
 generating the platform-agnostic query in a predetermined platform-agnostic language.