System and method for inference generation via optimization of inference model portions

Abstract

Methods and systems for inference generation are disclosed. To manage inference generation, a system may include an inference model manager and any number of data processing systems. The inference model manager may represent an inference model as a bipartite graph. To obtain portions of the inference model, the bipartite graph may be partitioned into portions and the portions may undergo an optimization process. The optimization process may include adding and/or competing for nodes of the bipartite graph in order to increase the stability of the portion with respect to the available computing resources of a corresponding data processing system. The optimization process may continue until all portions achieve stability in a way that reduces necessary communications between the data processing systems. Each portion of the inference model may be distributed to one data processing system so that the data processing systems may collectively generate inferences usable by a downstream consumer.

What is claimed is: 
     
       1. A method for executing an inference model across multiple data processing systems that each individually have insufficient computing resources to complete timely execution of the inference model, the method comprising:
 identifying available computing resources of each of the data processing systems; 
 dividing the inference models into portions based on:
 characteristics of units of the inference model that indicate data dependencies between the units of the inference model, and 
 an optimization process that groups the units into the portions of the inference model based on a likelihood of potential groups of the units exceeding the available computing resources of corresponding data processing systems; 
 
 distributing the portions of the inference model to the data processing systems; and 
 executing the inference model using the portions of the inference model distributed to the data processing systems to obtain an inference model result. 
 
     
     
       2. The method of  claim 1 , wherein the units of the inference model are neurons of a trained neural network. 
     
     
       3. The method of  claim 2 , wherein the data dependencies between the units of the inference model comprise weights for the corresponding neurons. 
     
     
       4. The method of  claim 1 , further comprising:
 obtaining a bipartite graph representation of the inference model. 
 
     
     
       5. The method of  claim 4 , wherein a first group of nodes of the bipartite graph representation correspond to neurons of a trained neural network and a second group of the nodes of the bipartite graph representation correspond to the data dependencies between the neurons of the trained neural network. 
     
     
       6. The method of  claim 5 , where performing the optimization process comprises:
 obtaining initial portions of the bipartite graph, with each of the initial portions comprising a node of the bipartite graph corresponding to a unit of the trained neural network; 
 optimizing the initial portions of the bipartite graph to obtain revised portions of the bipartite graph; and 
 adding units of the trained neural network to the portions of the inference model based on the revised portions of the bipartite graph. 
 
     
     
       7. The method of  claim 6 , wherein the initial portions of the bipartite graph are obtained by randomly assigning a node from the bipartite graph to each respective portion of the portions of the bipartite graph. 
     
     
       8. The method of  claim 7 , where optimizing the initial portions of the bipartite graph comprises:
 for a portion of the portions of the bipartite graph:
 performing a breadth-first search to identify a neighboring node of the bipartite graph with respect to a last added node of the portion; 
 adding the neighboring node to the portion to obtain a revised portion; 
 making a determination regarding whether the revised portion is unstable; 
 in an instance where the determination indicates that the portion is unstable:
 dividing the revised portion into two portions to obtain a set of stable portions; and 
 
 in an instance where the determination indicates that the portion is stable:
 retaining the revised portion to obtain the set of stable portions. 
 
 
 
     
     
       9. The method of  claim 8 , where optimizing the initial portions of the bipartite graph further comprises:
 revising the set of stable portions based on a competitive process that removes nodes from portions of the set of stable portions that are closer to a stability limit and adding the removed nodes to other portions of the set of stable portions that are further away from the stability limit. 
 
     
     
       10. The method of  claim 8 , wherein the neighboring node has a data dependency with respect to the last added node. 
     
     
       11. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for executing an inference model across multiple data processing systems that each individually have insufficient computing resources to complete timely execution of the inference model, the operations comprising:
 identifying available computing resources of each of the data processing systems; 
 dividing the inference models into portions based on:
 characteristics of units of the inference model that indicate data dependencies between the units of the inference model, and 
 an optimization process that groups the units into the portions of the inference model based on a likelihood of potential groups of the units exceeding the available computing resources of corresponding data processing systems; 
 
 distributing the portions of the inference model to the data processing systems; and 
 executing the inference model using the portions of the inference model distributed to the data processing systems to obtain an inference model result. 
 
     
     
       12. The non-transitory machine-readable medium of  claim 11 , wherein the units of the inference model are neurons of a trained neural network. 
     
     
       13. The non-transitory machine-readable medium of  claim 12 , wherein the data dependencies between the units of the inference model comprise weights for the corresponding neurons. 
     
     
       14. The non-transitory machine-readable medium of  claim 11 , wherein the operations further comprise:
 obtaining a bipartite graph representation of the inference model. 
 
     
     
       15. The non-transitory machine-readable medium of  claim 14 , wherein a first group of nodes of the bipartite graph representation correspond to neurons of a trained neural network and a second group of the nodes of the bipartite graph representation correspond to the data dependencies between the neurons of the trained neural network. 
     
     
       16. A data processing system, comprising:
 a processor; and 
 a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for executing an inference model across multiple data processing systems that each individually have insufficient computing resources to complete timely execution of the inference model, the operations comprising:
 identifying available computing resources of each of the data processing systems; 
 dividing the inference models into portions based on:
 characteristics of units of the inference model that indicate data dependencies between the units of the inference model, and 
 an optimization process that groups the units into the portions of the inference model based on a likelihood of potential groups of the units exceeding the available computing resources of corresponding data processing systems; 
 
 distributing the portions of the inference model to the data processing systems; and 
 executing the inference model using the portions of the inference model distributed to the data processing systems to obtain an inference model result. 
 
 
     
     
       17. The data processing system of  claim 16 , wherein the units of the inference model are neurons of a trained neural network. 
     
     
       18. The data processing system of  claim 17 , wherein the data dependencies between the units of the inference model comprise weights for the corresponding neurons. 
     
     
       19. The data processing system of  claim 16 , wherein the operations further comprise:
 obtaining a bipartite graph representation of the inference model. 
 
     
     
       20. The data processing system of  claim 19 , wherein a first group of nodes of the bipartite graph representation correspond to neurons of a trained neural network and a second group of the nodes of the bipartite graph representation correspond to the data dependencies between the neurons of the trained neural network.