Holistic task scheduling for distributed computing

Abstract

Embodiments of the present invention provide a method, system and computer program product for holistic task scheduling in a distributed computing environment. In an embodiment of the invention, a method for holistic task scheduling in a distributed computing environment is provided. The method includes selecting a first task for a first job and a second task for a different, second job, both jobs being scheduled for processing within a node a distributed computing environment by a task scheduler executing in memory by at least one processor of a computer. The method also can include comparing an estimated time to complete the first and second jobs. Finally, the first task can be scheduled for processing in the node when the estimated time to complete the second job exceeds the estimated time to complete the first job. Otherwise the second task can be scheduled for processing in the node when the estimated time to complete the first job exceeds the estimated time to complete the second job.

1 . A method for holistic task scheduling in a distributed computing environment, the method comprising:
 selecting first and second tasks associated with respectively different first and second jobs scheduled for processing within a node a distributed computing environment by a task scheduler executing in memory by at least one processor of a computer;   comparing an estimated time to complete each of the first and second jobs; and,   scheduling the first task for processing in the node when the estimated time to complete the second job exceeds the estimated time to complete the first job, but otherwise scheduling the second task for processing in the node when the estimated time to complete the first job exceeds the estimated time to complete the second job.   
     
     
         2 . The method of  claim 1 , wherein the first and second jobs individually comprise a set of tasks for a sub-problem for a problem processed according to map/reduce. 
     
     
         3 . The method of  claim 1 , wherein the distributed computing environment is a cloud computing cluster. 
     
     
         4 . The method of  claim 1 , wherein scheduling the first task for processing in the node when the estimated time to complete the second job exceeds the estimated time to complete the first job, but otherwise scheduling the second task for processing in the node when the estimated time to complete the first job exceeds the estimated time to complete the second job comprises, scheduling the second task for processing in the node when the estimated time to complete the second job exceeds the estimated time to complete the first job but a time required to switch from executing tasks associated with the second job to tasks from the first job exceeds a threshold value, and otherwise scheduling the first task for processing in the node. 
     
     
         5 . A task scheduling data processing system comprising:
 a computer with at least one processor and memory and coupled to a distributed computing system of different nodes over a computer communications network;   a parallel processing module configured to perform parallel processing of a provided problem in the different nodes; and,   a task scheduler executing in the computer, the task scheduler comprising program code that when executed in the memory of the computer is enabled to select first and second tasks associated with respectively different first and second jobs scheduled for processing within a node of the distributed computing system by the task scheduler, to compare an estimated time to complete the first and second jobs, and to schedule the first task for processing in the node when the estimated time to complete the second job exceeds the estimated time to complete the first job, but otherwise scheduling the second task for processing in the node when the estimated time to complete the first job exceeds the estimated time to complete the second job.   
     
     
         6 . The system of  claim 5 , wherein the parallel processing module is a map/reduce module configured to perform a map/reduce operation on a selected problem. 
     
     
         7 . The system of  claim 4 , wherein the distributed computing system is a cloud computing cluster. 
     
     
         8 . A computer program product for holistic task scheduling in a distributed computing environment, the computer program product comprising:
 a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:   computer readable program code for selecting first and second tasks associated with respectively different first and second jobs scheduled for processing within a node a distributed computing environment by a task scheduler executing in memory by at least one processor of a computer;   computer readable program code for comparing an estimated time to complete the first and second jobs; and,   computer readable program code for scheduling the first task for processing in the node when the estimated time to complete the second job exceeds the estimated time to complete the first job, but otherwise scheduling the second task for processing in the node when the estimated time to complete the first job exceeds the estimated time to complete the second job.   
     
     
         9 . The computer program product of  claim 8 , wherein the first and second jobs individually comprise a set of tasks for a sub-problem for a problem processed according to map/reduce. 
     
     
         10 . The computer program product of  claim 8 , wherein the distributed computing environment is a cloud computing cluster. 
     
     
         11 . The computer program product of  claim 8 , wherein the computer readable program code for scheduling the first task for processing in the node when the estimated time to complete the second job exceeds the estimated time to complete the first job, but otherwise scheduling the second task for processing in the node when the estimated time to complete the first job exceeds the estimated time to complete the second job comprises, computer readable program code for scheduling the second task for processing in the node when the estimated time to complete the second job exceeds the estimated time to complete the first job but a time required to switch from executing tasks associated with the second job to tasks from the first job exceeds a threshold value, and otherwise scheduling the first task for processing in the node.