Systems, methods, and devices for dynamic resource monitoring and allocation in a cluster system

Abstract

In an embodiment, the systems, methods, and devices disclosed herein comprise a computer resource monitoring and allocation system. In an embodiment, the resource monitoring and allocation system can be configured to allocate computer resources that are available on various nodes of a cluster to specific jobs and/or sub-jobs and/or tasks and/or processes.

1 . (canceled) 
     
     
         2 . (canceled) 
     
     
         3 . (canceled) 
     
     
         4 . (canceled) 
     
     
         5 . (canceled) 
     
     
         6 . (canceled) 
     
     
         7 . (canceled) 
     
     
         8 . (canceled) 
     
     
         9 . (canceled) 
     
     
         10 . (canceled) 
     
     
         11 . (canceled) 
     
     
         12 . (canceled) 
     
     
         13 . (canceled) 
     
     
         14 . (canceled) 
     
     
         15 . (canceled) 
     
     
         16 . (canceled) 
     
     
         17 . (canceled) 
     
     
         18 . (canceled) 
     
     
         19 . (canceled) 
     
     
         20 . (canceled) 
     
     
         21 . A non-virtual computer cluster system, the system comprising:
 a management computing device comprising one or more computer readable storage devices configured to store a plurality of computer executable instructions and one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the plurality of computer executable instructions in order to cause the management computing device to coordinate processing of a plurality of sub-jobs for an overall job; and   a plurality of computer system nodes configured to perform processing of the plurality of sub-jobs for the overall job, each of the plurality of computer system nodes comprising one or more computer readable storage devices configured to store a plurality of computer executable instructions and one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the plurality of computer executable instructions in order to cause each of the plurality of computer system nodes to:
 monitor utilization of system resources of the computer system node for processing one or more of the plurality of sub-jobs, the system resources comprising at least one of CPU, electronic storage input/output, network, and memory of the computer system node; and 
 periodically report the monitored system resource utilization to the management computing device, 
   wherein the management computing device is a separate computing device from each of the plurality of computer system nodes, and   wherein the management computing device is configured to coordinate processing of the plurality of sub-jobs by the plurality of computer system nodes based at least in part on the monitored system resource utilization of the plurality of computer system nodes.   
     
     
         22 . The non-virtual computer cluster system of  claim 21 , wherein the management computing device is further configured to determine progress for completing the overall job by tracking the plurality of sub-jobs being processed by the plurality of computer system nodes. 
     
     
         23 . The non-virtual computer cluster system of  claim 21 , wherein the management computing device is further configured to dynamically increase processing of higher priority sub-jobs. 
     
     
         24 . The non-virtual computer cluster system of  claim 21 , wherein the management computing device operates a master node of the computer cluster. 
     
     
         25 . The non-virtual computer cluster system of  claim 21 , wherein the management computing device is further configured to dynamically adjust system resource allocations for each sub-job on the plurality of computer system nodes. 
     
     
         26 . The non-virtual computer cluster system of  claim 25 , wherein the dynamic adjustment of system resource allocations is based at least in part on the periodic reporting from the plurality of computer system nodes. 
     
     
         27 . The non-virtual computer cluster system of  claim 26 , wherein the dynamic adjustment of system resource allocations comprises allocating additional central processing unit resources to a sub-job operating on a computer system node based at least in part on the management computing device determining that a total amount of unused central processing unit resources available at the computer system node exceeds a threshold level. 
     
     
         28 . The non-virtual computer cluster system of  claim 27 , wherein the dynamic adjustment of system resource allocations comprises allocating additional networking resources to a sub-job operating on a computer system node based at least in part on the management computing device determining that a total amount of unused networking resources available at the computer system node exceeds a threshold level. 
     
     
         29 . The non-virtual computer cluster system of  claim 27 , wherein the dynamic adjustment of system resource allocations comprises allocating additional random-access memory resources to a sub-job operating on a computer system node based at least in part on the management computing device determining that a total amount of unused random-access memory resources available at the computer system node exceeds a threshold level. 
     
     
         30 . The non-virtual computer cluster system of  claim 27 , wherein the dynamic adjustment of system resource allocations comprises allocating additional electronic storage input-output resources to a sub-job operating on a computer system node based at least in part on the management computing device determining that a total amount of unused electronic storage input-output resources available at the computer system node exceeds a threshold level. 
     
     
         31 . The non-virtual computer cluster system of  claim 21 , wherein the management computing device is further configured to receive goals specified by an operator of the computer cluster system, wherein the goals comprise a priority level for one or more of the plurality of sub-jobs. 
     
     
         32 . The non-virtual computer cluster system of  claim 31 , wherein the management computing device is configured to assign a first level of system resource allocations to a first sub-job and a second level of system resource allocations to a second sub-job, wherein the first level is greater than the second level, wherein the first sub-job comprises a first priority level and the second sub-job comprises a second priority level, wherein the first priority level is greater than the second priority level. 
     
     
         33 . The non-virtual computer cluster system of  claim 31 , wherein the goals specified by the operator of the computer cluster comprises a user group preference for one or more of the plurality of sub-jobs. 
     
     
         34 . The non-virtual computer cluster system of  claim 33 , wherein the management computing device is configured to assign a first level of system resource allocations to a first sub-job and a second level of system resource allocations to a second sub-job, wherein the first level is greater than the second level, wherein the first sub-job comprises a first user group preference and the second sub-job comprises a second user group preference, wherein sub-jobs associated with the first user group preference are processed with greater priority than sub-jobs associated with the second user group preference. 
     
     
         35 . A computer-implemented method for coordinating processing of a plurality of sub-jobs for an overall job by a plurality of computer system nodes in a non-virtual computer cluster system, the method comprising:
 monitoring, by each of the plurality of computer system nodes of the computer cluster system, utilization of system resources of each of the plurality of computer system nodes for processing one or more of the plurality of sub-jobs for the overall job, the system resources comprising at least one of CPU, electronic storage input/output, network, and memory of the particular computer system node;   periodically reporting, by each of the plurality of computer system nodes of the computer cluster system, the monitored system resource utilization to the management computing device of the computer cluster system;   receiving, by the management computing device, the monitored system resource utilization of each of the plurality of computer system nodes of the computer cluster system; and   coordinating, by the management computing device, processing of the plurality of sub-jobs by the plurality of computer system nodes based at least in part on the monitored system resource utilization of the plurality of computer system nodes,   wherein each of the plurality of computer system nodes and the management computing device comprises a computer processor and an electronic storage medium.   
     
     
         36 . The computer-implemented method of  claim 35  further comprising determining, by the management computing device, progress for completing the overall job by tracking the plurality of sub-jobs being processed by the plurality of computer system nodes. 
     
     
         37 . The computer-implemented method of  claim 35  further comprising receiving, by the management computing device, a priority level for one or more of the plurality of sub-jobs from an operator of the computer cluster system. 
     
     
         38 . The computer-implemented method of  claim 37  further comprising assigning, by the management computing device, a first level of system resource allocations to a first sub-job and a second level of system resource allocations to a second sub-job, wherein the first level is greater than the second level, wherein the first sub-job comprises a first priority level and the second sub-job comprises a second priority level, wherein the first priority level is greater than the second priority level. 
     
     
         39 . The computer-implemented method of  claim 35 , further comprising dynamically adjusting, by the management computing device, system resource allocations for one or more of the plurality of sub-jobs being processed by the plurality of computer system nodes. 
     
     
         40 . The computer-implemented method of  claim 39 , wherein the dynamic adjustment of system resource allocations is based at least in part on the periodic reporting from the plurality of computer system nodes.