Systems, methods, and devices for dynamic resource monitoring and allocation in a cluster system

Abstract

In an embodiment, the systems, methods, and devices disclosed herein comprise a computer resource monitoring and allocation system. In an embodiment, the resource monitoring and allocation system can be configured to allocate computer resources that are available on various nodes of a cluster to specific jobs and/or sub-jobs and/or tasks and/or processes.

1 . (canceled) 
     
     
         2 . (canceled) 
     
     
         3 . (canceled) 
     
     
         4 . (canceled) 
     
     
         5 . (canceled) 
     
     
         6 . (canceled) 
     
     
         7 . (canceled) 
     
     
         8 . (canceled) 
     
     
         9 . (canceled) 
     
     
         10 . (canceled) 
     
     
         11 . (canceled) 
     
     
         12 . (canceled) 
     
     
         13 . (canceled) 
     
     
         14 . (canceled) 
     
     
         15 . (canceled) 
     
     
         16 . (canceled) 
     
     
         17 . (canceled) 
     
     
         18 . (canceled) 
     
     
         19 . (canceled) 
     
     
         20 . (canceled) 
     
     
         21 . A Hadoop computer system comprising:
 a first Hadoop node comprising one or more processors and configured to perform computing processes on tasks assigned to the first Hadoop node;   a second Hadoop node comprising one or more processors and configured to perform computing processes on tasks assigned to the second Hadoop node;   a resource controller configured to receive a computing job submission, wherein the resource controller is further configured to divide the received computing job submission into a first task and a second task, and to assign the first task to the first Hadoop node and to assign the second task to the second Hadoop node;   the first Hadoop node comprising a first node controller, the first node controller configured to transmit first node memory information that includes data indicating memory resource availability of the first Hadoop node; and   the second Hadoop node comprising a second node controller, the second node controller configured to transmit second node memory information that includes data indicating memory resource availability of the second Hadoop node,   wherein the resource controller is configured to receive the first node memory information from the first Hadoop node and, based at least in part on the data indicating memory resource availability of the first Hadoop node, to determine whether to increase memory for use by tasks of the first Hadoop node, and whether to decrease memory for use by tasks of the first Hadoop node; and   wherein the resource controller is configured to receive the second node memory information from the second Hadoop node and, based at least in part on the data indicating memory resource availability of the second Hadoop node, to determine whether to increase memory for use by tasks of the second Hadoop node, and whether to decrease memory for use by tasks of the second Hadoop node.   
     
     
         22 . The Hadoop computer system of  claim 21 , wherein the assignment of the first task occurs after the determination to increase memory for use by tasks of the first Hadoop node. 
     
     
         23 . The Hadoop computer system of  claim 21 , wherein the resource controller is further configured to increase memory for use by tasks of the first Hadoop node by increasing the number of containers available on the first Hadoop node. 
     
     
         24 . The Hadoop computer system of  claim 21 , wherein the resource controller is further configured to decrease the amount of memory utilized by the first Hadoop node by instructing to kill one or more processes corresponding to the first task assigned to the first Hadoop node. 
     
     
         25 . The Hadoop computer system of  claim 21 , wherein the resource controller is further configured to decrease the amount of memory utilized by the first Hadoop node by instructing to preempt one or more processes corresponding to the first task assigned to the first Hadoop node. 
     
     
         26 . The Hadoop computer system of  claim 21 , wherein the resource controller is further configured to decrease the amount of memory utilized by the first Hadoop node by instructing to relocate one or more processes corresponding to the first task assigned to the first Hadoop node. 
     
     
         27 . The Hadoop computer system of  claim 21 , wherein the repeated transmission of the first network transmission and the repeated transmission of the second network transmission are performed periodically. 
     
     
         28 . The Hadoop computer system of  claim 21 , wherein the repeated transmission of the first network transmission and the repeated transmission of the second network transmission are performed in substantially real-time. 
     
     
         29 . The Hadoop computer system of  claim 21 , wherein the first node controller is further configured to repeatedly transmit the first network transmission comprising data indicating memory resource availability of the first Hadoop node directly to the resource controller, and wherein the second node controller is further configured to repeatedly transmit the second network transmission comprising data indicating memory resource availability of the second Hadoop node directly to the resource controller. 
     
     
         30 . The Hadoop computer system of  claim 21 , wherein the first Hadoop node further comprises a Hadoop Node Manager, wherein the first node controller is part of a same process as the Hadoop Node Manager of the first Hadoop node, wherein the second Hadoop node further comprises a Hadoop Node Manager, and wherein the second node controller is part of a same process as the Hadoop Node Manager of the second Hadoop node. 
     
     
         31 . The Hadoop computer system of  claim 21 , wherein the first Hadoop node further comprises a Hadoop Node Manager, wherein the first node controller is part of a separate process from the Hadoop Node Manager of the first Hadoop node, wherein the second Hadoop node further comprises a Hadoop Node Manager, and wherein the second node controller is part of a separate process from the Hadoop Node Manager of the second Hadoop node. 
     
     
         32 . The Hadoop computer system of  claim 21 , further comprising a Hadoop Resource Manager, wherein the Hadoop Resource Manager is part of a same process as the resource controller. 
     
     
         33 . The Hadoop computer system of  claim 21 , wherein the memory resource availability of the first Hadoop node is determined based on memory resources available for assignment of one or more additional containers on the first Hadoop node, and wherein the memory resource availability of the second Hadoop node is determined based on memory resources available for assignment of one or more additional containers on the second Hadoop node. 
     
     
         34 . The Hadoop computer system of  claim 21 , wherein the memory resource availability of the first Hadoop node is determined based on an availability of containers assigned to the first Hadoop node, and wherein the memory resource availability of the second Hadoop node is determined based on an availability of containers assigned to the second Hadoop node. 
     
     
         35 . The Hadoop computer system of  claim 21 , wherein the memory resource availability of the first Hadoop node is determined based on an amount of random access memory used by one or more processes on the first Hadoop node, and wherein the memory resource availability of the second Hadoop node is determined based on an amount of random access memory used by one or more processes on the second Hadoop node. 
     
     
         36 . The Hadoop computer system of  claim 21 , wherein the memory resource availability of the first Hadoop node is determined based on an amount of random access memory reported as being used by an operating system of the first Hadoop node, and wherein the memory resource availability of the second Hadoop node is determined based on an amount of random access memory reported as being used by an operating system of the second Hadoop node. 
     
     
         37 . The Hadoop computer system of  claim 21 , wherein the memory resource availability of the first Hadoop node is determined based on an unused amount of random access memory on the first Hadoop node, and wherein the memory resource availability of the second Hadoop node is determined based on an unused amount of random access memory on the second Hadoop node. 
     
     
         38 . The Hadoop computer system of  claim 21 , wherein the resource controller is further configured to repeatedly receive data indicating processing resource availability of the first Hadoop node and data indicating processing resource availability of the second Hadoop node. 
     
     
         39 . The Hadoop computer system of  claim 21 , wherein the resource controller is further configured to repeatedly receive data indicating storage drive resource availability of the first Hadoop node and data indicating storage drive resource availability of the second Hadoop node. 
     
     
         40 . The Hadoop computer system of  claim 21 , wherein the resource controller is further configured to repeatedly receive data indicating network activity resource availability of the first Hadoop node and data indicating network activity resource availability of the second Hadoop node.