Dynamic run time allocation of distributed jobs

Abstract

A job optimizer dynamically changes the allocation of processing units on a multi-nodal computer system. A distributed application is organized as a set of connected processing units. The arrangement of the processing units is dynamically changed at run time to optimize system resources and interprocess communication. A collector collects metrics of the system, nodes, application, jobs and processing units that will be used to determine how to best allocate the jobs on the system. A job optimizer analyzes the collected metrics to dynamically arrange the processing units within the jobs. The job optimizer may determine to combine multiple processing units into a job on a single node when there is an overutilization of interprocess communication between processing units. Alternatively, the job optimizer may determine to split a job's processing units into multiple jobs on different nodes where the processing units are over utilizing the resources on the node.

1 . An apparatus comprising:
 a) a plurality of nodes of a multi-nodal computer system, wherein the plurality of nodes are connected by a plurality of networks, where each of the plurality of nodes has at least one central processing unit (CPU) coupled to a memory;   b) an application having a plurality of jobs, each with at least one processing unit executing on the plurality of nodes;   c) a collector collecting metrics of the system, nodes, application, jobs and processing units in order to determine how to best allocate the jobs on the system; and   d) a job optimizer dynamically changing the allocation of processing units on the plurality of nodes based on the collected metrics.   
     
     
         2 . The apparatus of  claim 1  wherein the job optimizer dynamically changes the allocation of the processing units by combining at least two processing units from jobs on different nodes into a job on a single node of the plurality of nodes. 
     
     
         3 . The apparatus of  claim 1  wherein the job optimizer dynamically changes the allocation of the processing units by splitting a job into multiple jobs on different nodes of the plurality of nodes. 
     
     
         4 . The apparatus of  claim 1  wherein the job optimizer dynamically changes the allocation of the processing units by splitting a job into multiple jobs on a same node to utilize multiple processors of the same node. 
     
     
         5 . The apparatus of  claim 1  wherein the metrics associated with the processing unit are selected from: CPU utilization by the processing unit, memory utilization by the processing unit, data throughput of the processing unit, and data latency for the processing unit. 
     
     
         6 . The apparatus of  claim 1  wherein the metrics associated with the application are selected from: aggregate CPU utilization by the application, aggregate memory utilization by the application, a result throughput for the application, and a result latency for the application. 
     
     
         7 . The apparatus of  claim 1  wherein the metrics associated with the node are selected from: CPU utilization for the node, memory utilization for the node and heap size for the node. 
     
     
         8 . The apparatus of  claim 1  wherein the metrics associated with the computer system are selected from: aggregate CPU utilization across the multi-nodal system, aggregate memory utilization across the multi-nodal system, aggregate network load across the multi-nodal system, and node-to-node network utilization. 
     
     
         9 . A computer implemented method for dynamically changing the allocation of processing units on a multi-nodal computer system comprising the steps of:
 a) executing an application having a plurality jobs, each with at least one processing unit on a plurality of nodes, where each node has at least one central processing unit (CPU) and a memory;   b) collecting metrics associated with the multi-nodal computer system, the application, the jobs, the plurality of nodes and the processing units;   c) analyzing the metrics;   d) when there is an over utilized resource performing the steps of:
 identifying jobs affecting the over utilized resource; 
 assessing potential job and processing unit permutations; and 
 dynamically changing the allocation of the processing units on the compute nodes based on the collected metrics. 
   
     
     
         10 . The computer implemented method of  claim 9  wherein the step of dynamically changing the allocation of the processing units further comprises combining at least two processing units into a job on a single node. 
     
     
         11 . The computer implemented method of  claim 9  wherein the step of dynamically changing the allocation of the processing units further comprises splitting a job into multiple jobs on a plurality of nodes. 
     
     
         12 . The computer implemented method of  claim 9  wherein the metrics associated with the processing unit are selected from: CPU utilization by the processing unit, memory utilization by the processing unit, data throughput of the processing unit, and data latency for the processing unit. 
     
     
         13 . The computer implemented method of  claim 9  wherein the metrics associated with the application are selected from: aggregate CPU utilization by the application, aggregate memory utilization by the application, a result throughput for the application, and a result latency for the application. 
     
     
         14 . The computer implemented method of  claim 9  wherein the metrics associated with the computer system are selected from: aggregate CPU utilization across the multi-nodal system, aggregate memory utilization across the multi-nodal system, aggregate network load across the multi-nodal system, and node-to-node network utilization. 
     
     
         15 . A computer implemented method for dynamically changing the allocation of processing units on a multi-nodal computer system comprising the steps of:
 a) executing an application having a plurality of jobs, each with at least one processing unit on a plurality of nodes, where each node has at least one central processing unit (CPU) and a memory;   b) collecting metrics associated with the multi-nodal computer system, the application, the jobs, the plurality of nodes and the processing units;   c) analyzing the metrics;   d) where there is an over utilized resource perform the steps of:   e) identifying jobs affecting the over utilized resource;   f) assessing potential job and processing unit permutations;   g) dynamically changing the allocation of the processing units on the plurality of compute nodes based on the collected metrics by combining at least two processing units into a job on a single node, and further dynamically changing the allocation of the processing units by splitting a job into multiple jobs on a plurality of nodes;   wherein the metrics associated with the processing unit are selected from: CPU utilization by the processing unit, memory utilization by the processing unit, data throughput of the processing unit, and data latency for the processing unit;   wherein the metrics associated with the application are selected from: aggregate CPU utilization by the application, aggregate memory utilization by the application, a result throughput for the application, and a result latency for the application;   wherein the metrics associated with the node are selected from: CPU utilization for the node and memory utilization for the node; and   wherein the metrics associated with the computer system are selected from: aggregate CPU utilization across the multi-nodal system, aggregate memory utilization across the multi-nodal system, aggregate network load across the multi-nodal system, and node-to-node network utilization.   
     
     
         16 . An article of manufacture comprising software stored on a computer-readable storage medium comprising:
 a collector for collecting metrics of a multi-nodal computer system, a plurality of nodes, an application, a plurality of jobs each with at least one processing unit, where the metrics are collected in order to determine how to best allocate the jobs on the system; and   a job optimizer that analyzes the collected metrics and dynamically changing the allocation of processing units on the plurality of nodes based on the collected metrics.   
     
     
         17 . The article of manufacture of  claim 16  wherein the job optimizer dynamically changes the allocation of the processing units by combining at least two processing units from jobs on different nodes into a job on a single node of the plurality of nodes. 
     
     
         18 . The article of manufacture of  claim 16  wherein the job optimizer dynamically changes the allocation of the processing units by splitting a job into multiple jobs on different nodes of the plurality of nodes. 
     
     
         19 . The article of manufacture of  claim 16  wherein the metrics associated with the processing unit are selected from: CPU utilization by the processing unit, memory utilization by the processing unit, data throughput of the processing unit, and data latency for the processing unit. 
     
     
         20 . The article of manufacture of  claim 16  wherein the metrics associated with the application are selected from: aggregate CPU utilization by the application, aggregate memory utilization by the application, a result throughput for the application, and a result latency for the application. 
     
     
         21 . The article of manufacture of  claim 16  wherein the metrics associated with the node are selected from: CPU utilization for the node and memory utilization for the node. 
     
     
         22 . The article of manufacture of  claim 16  wherein the metrics associated with the computer system are selected from: aggregate CPU utilization across the multi-nodal system, aggregate memory utilization across the multi-nodal system, aggregate network load across the multi-nodal system, and node-to-node network utilization.