Automated System For Generative Multimodel Multiclass Classification And Similarity Analysis Using Machine Learning

Abstract

A sample of data is placed within a directed graph that comprises a plurality of hierarchical nodes that form a queue of work items for a particular worker class that are used to process the sample of data. Subsequently, work items are scheduled within the queue for each of a plurality of workers by traversing the nodes of the directed graph. The work items are then served to the workers according to the queue. Results can later be received from the workers for the work items (the nodes of the directed graph are traversed based on the received results). In addition, in some variations, the results can be classified so that one or models can be generated. Related systems, methods, and computer program products are also described.

What is claimed is: 
     
         1 . A method for implementation by one or more data processors forming part of at least one computing system, the method comprising:
 placing a sample of data within a directed graph that comprises a plurality of hierarchical nodes that form a queue of work items for a particular worker class that is used to process the sample of data;   scheduling work items within the queue for each of a plurality of workers by traversing the nodes of the directed acyclic graph;   serving the work items to the workers according to the queue; and   receiving results from the workers for the work items;   wherein the nodes of the directed graph are traversed based on the received results.   
     
     
         2 . The method of  claim 1  wherein the results comprise extracted features from the sample of data. 
     
     
         3 . The method of  claim 2  further comprising:
 classifying the sample of data and/or the extracted features. 
 
     
     
         4 . The method of  claim 3  further comprising:
 generating at least one model using the extracted features and the classification. 
 
     
     
         5 . The method of  claim 1  further comprising:
 classifying the sample of data based on the received results. 
 
     
     
         6 . The method of  claim 5  further comprising:
 providing data characterizing the classifying, the providing comprising at least one of: displaying the data characterizing the classifying, storing the data characterizing the classifying, loading the data characterizing the classifying into memory, or transmitting the data characterizing the classifying to a remote computing system. 
 
     
     
         7 . The method of  claim 1 , wherein the results further comprise routing data that is used to determine where to schedule a next subsequent work item in the queue. 
     
     
         8 . The method of  claim 1  further comprising:
 prioritizing an order of each sample prior to adding such samples to the queue, wherein each sample is added to the queue according to the prioritized order. 
 
     
     
         9 . The method of  claim 8 , wherein the priorities are based on a pre-defined rate of processing. 
     
     
         10 . The method of  claim 8 , wherein prioritization of the at least one sample is locally adjusted in real-time. 
     
     
         11 . The method of  claim 1 , wherein the work items are scheduled in the queue according to at least one of sample prioritization or worker rate. 
     
     
         12 . The method of  claim 1 , wherein the workers to which the work items are served are part of a pool having a dynamically changing size based on available resources. 
     
     
         13 . The method of  claim 12 , wherein the available resources are based on determined supply and demand. 
     
     
         14 . The method of  claim 5 , wherein the sample of data comprises files for access or execution by a computing system, and wherein the classification indicates whether or not at least one file likely comprises malicious code. 
     
     
         15 . The method of  claim 5 , wherein the sample of data comprises medical imaging data and wherein the classification indicated whether or not at least one portion of the medical imaging data indicates a likelihood of an abnormal condition. 
     
     
         16 . The method of  claim 1 , wherein the directed graph is a directed acyclic graph. 
     
     
         17 . A non-transitory computer program product storing instructions which, when executed by at least one data processor forming part of at least one computing system, result in operations comprising:
 placing a sample of data within a directed graph that comprises a plurality of hierarchical nodes that form a queue of work items for a particular worker class that is used to process the sample of data;   scheduling work items within the queue for each of a plurality of workers by traversing the nodes of the directed acyclic graph;   serving the work items to the workers according to the queue;   receiving results from the workers for the work items comprising extracted features; and   classifying at least a portion of the extracted features using one or more machine learning models;   wherein the nodes of the directed acyclic graph are traversed based on the received results.   
     
     
         18 . The computer program product of  claim 17 , wherein the sample of data comprises files for access or execution by a computing system, and wherein the classification indicates whether or not at least one file likely comprises malicious code. 
     
     
         19 . The computer program product of  claim 17 , wherein the sample of data comprises medical imaging data and wherein the classification indicated whether or not at least one portion of the medical imaging data indicates a likelihood of an abnormal condition. 
     
     
         20 . A system comprising:
 at least one data processor; and   memory storing instructions which, when executed by the at least one data processor, result in operations comprising:
 placing a sample of data within a directed acyclic graph, the directed acyclic graph comprising a plurality of hierarchical nodes that form a queue of work items for a particular worker class that is used to process the sample of data; 
 scheduling work items within the queue for each of a plurality of workers by traversing the nodes of the directed acyclic graph; 
 serving the work items to the workers according to the queue; 
 receiving results from the workers for the work items comprising extracted features; 
 classifying at least a portion of the extracted features; and 
 generating at least one machine learning model using the classified extracted features. 
   wherein the nodes of the directed acyclic graph are traversed based on the received results.   
     
     
         21 . The system of  claim 20 , wherein the sample of data comprises files for access or execution by a computing system, and wherein the classification indicates whether or not at least one file likely comprises malicious code. 
     
     
         22 . The system of  claim 20 , wherein the sample of data comprises medical imaging data and wherein the classification indicated whether or not at least one portion of the medical imaging data indicates a likelihood of an abnormal condition.