Method for processing data in b2b cloud distribution platform system

Abstract

A method for processing data in a B2B cloud distribution platform system, which includes dividing server nodes into primary cluster, secondary cluster and idle cluster pool; obtain the average resource utilization of multiple server nodes in the primary and secondary clusters on a regular basis. If the average resource utilization of the server nodes in the primary cluster is greater than or equal to the minimum threshold, the server nodes in the idle cluster pool are migrated to the primary cluster. If the average resource utilization of the server nodes in the secondary cluster is less than the minimum threshold, the server nodes in the secondary cluster are moved out to the idle cluster pool. If it is between the lowest and highest thresholds, it will not migrate. If it is greater than or equal to the maximum threshold, the server nodes in the idle cluster pool are migrated to the secondary cluster. Based on the problem that the existing cloud distribution platform system has a large amount of data processing and a high requirement for server resources. The present invention puts forward a load balancing method, which can make the best use of server resources and improve the data processing efficiency of the platform system.

1 . A method for processing data in a B2B cloud distribution platform system, comprising:
 Step S1: divide all server nodes into primary cluster, secondary cluster and idle cluster pool, there are multiple server nodes in the primary cluster, secondary cluster and idle cluster pool, the server nodes in the primary cluster are used to handle core business, and the server nodes in the secondary cluster are used to handle basic data business with low real-time requirements and large processing capacity, the idle cluster pool is used to accommodate idle server nodes;   Step S2: cloud distribution platform system periodically obtains the average resource utilization of multiple server nodes in the primary cluster and the secondary cluster by polling;   Step S3: if the average resource utilization of the server nodes in the primary cluster is greater than or equal to the minimum threshold, the server nodes in the idle cluster pool will be migrated to the primary cluster until the average resource utilization of the server nodes in the primary cluster is less than the minimum threshold, and then skip to Step S4; if the average resource utilization of the server nodes in the primary cluster is less than the minimum threshold, it jumps directly to Step S4;   Step S4: if the average resource utilization of the server nodes in the secondary cluster, the server nodes with the highest resource utilization in the primary cluster will be moved out of the idle cluster pool in order from large to small until the average resource utilization of the server nodes in the primary cluster is greater than or equal to the minimum threshold; if the average resource utilization of server nodes in the secondary cluster is between the minimum and maximum thresholds, no migration occurs, if the average resource utilization of the server nodes in the primary cluster is greater than or equal to the maximum threshold, the server nodes in the idle cluster pool are migrated to the secondary cluster until the average resource utilization of the server nodes in the secondary cluster is less than the maximum threshold.   
     
     
         2 . The method for processing data of  claim 1 , wherein step S3, if the average resource utilization of the server nodes in the primary cluster is greater than or equal to the minimum threshold, after migrating the server nodes in the idle cluster pool to the primary cluster, migrate the service directories in the original server nodes of the primary cluster to the incoming server nodes, in the order of resource utilization from large to small, prioritize migrating service directories in the server nodes with the highest resource utilization. 
     
     
         3 . The method for processing data of  claim 1 , wherein step S3, when migrating a service directory, determine if the size of the service directory you want to migrate is greater than the directory threshold; if the directory threshold is greater than the directory threshold, the migration directory is broken up into several sub-service directories that are less than or equal to the directory threshold, and the migration of the sub-service directories is completed in turn. 
     
     
         4 . The method for processing data of  claim 1 , wherein Step S4, if the average resource utilization of the server nodes in the secondary cluster is less than the minimum threshold, after the server nodes in the secondary cluster have been moved out, the directory of services on the moved server nodes will be moved into the remaining server nodes in the secondary cluster, and the server nodes with the least resource utilization will be moved in priority from small to large;
 if the average resource utilization of the server nodes in the secondary cluster is greater than or equal to the maximum threshold, then after migrating the server nodes in the idle cluster pool to the secondary cluster, the service directories in the original server nodes of the secondary cluster are migrated to the incoming server nodes in the order of resource utilization from large to small, prioritize migrating service directories in the server nodes with the highest resource utilization.   
     
     
         5 . The method for processing data of  claim 1 , wherein Step S4, when migrating a service directory, determine if the size of the service directory you want to migrate is greater than the directory threshold; if the directory threshold is greater than the directory threshold, the migration directory is broken up into several sub-service directories that are less than or equal to the directory threshold, and the migration of the sub-service directories is completed in turn. 
     
     
         6 . The method for processing data of  claim 1 , wherein the average resource utilization is calculated by averaging the resource utilization of all server nodes in the primary or secondary cluster, the resource utilization of each server node is obtained by weighted average of CPU utilization, memory utilization, network send and receive rate utilization. 
     
     
         7 . The method for processing data of  claim 1 , wherein Step S3, if the average resource utilization of the server nodes in the primary cluster is greater than or equal to the minimum threshold, after migrating the server nodes in the idle cluster pool to the primary cluster, load balancing of the server nodes in the primary cluster is achieved by following steps:
 Step S31: obtain the static and dynamic load parameters of each server node, and weighted average the comprehensive load parameters;   Step S32: the average comprehensive load parameters of the main cluster are obtained by averaging the comprehensive load parameters of all the server nodes, the server nodes whose comprehensive load parameters are lower than the average comprehensive load parameters are the target server nodes, and the server nodes whose comprehensive load parameters are higher than the average comprehensive load parameters are the source server nodes;   Step S33: source server nodes are arranged in the order of comprehensive load parameters from large to small, and target server nodes are arranged in the order of comprehensive load parameters from small to large, so that source server nodes and target server nodes correspond in the corresponding order; and   Step S34: in order from large to small, the service directory in the source server node with the largest comprehensive load parameter is migrated first to the target server node with the smallest comprehensive load parameter until all server nodes achieve load balancing.   
     
     
         8 . The method for processing data of  claim 7 , wherein Step S34, when migrating a service directory, determine if the size of the service directory you want to migrate is greater than the directory threshold; if the directory threshold is greater than the directory threshold, the migration directory is broken up into several sub-service directories that are less than or equal to the directory threshold, and the migration of the sub-service directories is completed in turn. 
     
     
         9 . The method for processing data of  claim 1 , wherein Step S4, if the average resource utilization of the server nodes in the secondary cluster is greater than or equal to the maximum threshold, the server nodes in the idle cluster pool are migrated to the secondary cluster, or, if the average utilization of server nodes in secondary cluster is less than the minimum threshold, load balancing of server nodes in secondary cluster is achieved by migrating the server nodes in the secondary cluster to the idle cluster pool and migrating the service directories on the migrated server nodes back to the secondary cluster by following steps:
 Step S41: obtain the static and dynamic load parameters of each server node, and weighted average the comprehensive load parameters;   Step S42: the average comprehensive load parameters of the secondary cluster are obtained by averaging the comprehensive load parameters of all the server nodes, the server nodes whose comprehensive load parameters are lower than the average comprehensive load parameters are the target server nodes, and the server nodes whose comprehensive load parameters are higher than the average comprehensive load parameters are the source server nodes;   Step S43: source server nodes are arranged in the order of comprehensive load parameters from large to small, and target server nodes are arranged in the order of comprehensive load parameters from small to large, so that source server nodes and target server nodes correspond in the corresponding order; and   Step S44: in order from large to small, the service directory in the source server node with the largest comprehensive load parameter is migrated first to the target server node with the smallest comprehensive load parameter until all server nodes achieve load balancing.   
     
     
         10 . The method for processing data of  claim 9 , wherein Step S44, when migrating a service directory, determine if the size of the service directory you want to migrate is greater than the directory threshold; if the directory threshold is greater than the directory threshold, the migration directory is broken up into several sub-service directories that are less than or equal to the directory threshold, and the migration of the sub-service directories is completed in turn.