Data processing system and method

Abstract

Embodiments of the present invention disclose a data processing apparatus. The apparatus is configured to: after calculating a set of gradient information of each parameter by using a sample data subset, delete the sample data subset, read a next sample data subset, calculate another set of gradient information of each parameter by using the next sample data subset, and accumulate a plurality of sets of calculated gradient information of each parameter, to obtain an update gradient of each parameter.

What is claimed is: 
     
         1 . A data processing apparatus, comprising:
 a gradient calculation module configured to:
 in a process of one iterative operation, sequentially read a plurality of sample data subsets from a sample data set, wherein each sample data subset comprises at least one piece of sample data;
 enter each read sample data subset into a machine learning model; and 
 calculate gradient information of each of a plurality of parameters of the machine learning model, wherein after a set of gradient information of each parameter is calculated using one sample data subset, the sample data subset is deleted before a next sample data subset is read, and another set of gradient information of each parameter is calculated using the next sample data subset, and wherein the machine learning model has an initialized global parameter or was updated in a last iterative operation; 
 
   an accumulation module configured to in the process of the one iterative operation, accumulate a plurality of sets of calculated gradient information of each parameter to obtain an update gradient of each parameter; and   a sending module configured to send the update gradient of each parameter in the process of the one iterative operation, wherein the update gradient of each parameter is used to update the machine learning model.   
     
     
         2 . The data processing apparatus according to  claim 1 , wherein the gradient calculation module, the accumulation module, and the sending module are further configured to participate in a plurality of iterative operations after the one iterative operation, until the machine learning model converges or a specified quantity of iterations are completed in calculation;
 in each of the plurality of iterative operations after the one iterative operation, the gradient calculation module, the accumulation module, and the sending module repeat actions in the process of the one iterative operation; and in the one iterative operation and the plurality of iterative operations after the one iterative operation, after the machine learning model is updated using an update gradient obtained in an iterative operation, a next iterative operation is performed.   
     
     
         3 . The data processing apparatus according to  claim 1 , wherein the accumulation module is configured to:
 for the plurality of sets of gradient information of each parameter that are obtained based on the plurality of read sample data subsets, accumulate a plurality of sets of gradient information of a same parameter to obtain the update gradient of each parameter.   
     
     
         4 . The data processing apparatus according to  claim 1 , wherein the accumulation module is configured to:
 for one set of gradient information of each parameter obtained based on each sample data subset, accumulate one set of gradient information of a same parameter to obtain an accumulation gradient of each parameter, so that a plurality of accumulation gradients of each parameter are obtained based on the plurality of read sample data subsets, and   accumulate the plurality of accumulation gradients of each parameter to obtain the update gradient of each parameter.   
     
     
         5 . The data processing apparatus according to  claim 1 , wherein the accumulation module is configured to:
 for the plurality of sets of gradient information of each parameter that are obtained based on the plurality of read sample data subsets, collect a plurality of sets of gradient information of a same parameter together, wherein the plurality of sets of gradient information of each parameter that are collected together are used as the update gradient of each parameter.   
     
     
         6 . The data processing apparatus according to  claim 1 , wherein the gradient calculation module is configured to:
 in the process of the one iterative operation, sequentially read the plurality of sample data subsets from the sample data set;   enter each read sample data subset into the machine learning model;   read and use an intermediate calculation result to calculate the gradient information of each of the plurality of parameters of the machine learning model, wherein the intermediate calculation result is used as input information to calculate the gradient information, and after a set of gradient information of each parameter is calculated using one sample data subset, the sample data subset is deleted before a next sample data subset is read, and another set of gradient information of each parameter is calculated using the next sample data subset; and   after the intermediate calculation result is used, delete the intermediate calculation result, wherein an operation of deleting the intermediate calculation result needs to be completed before the next sample data subset is read.   
     
     
         7 . A data processing method, wherein the method comprises one iterative operation, and the one iterative operation comprises:
 in a process of the one iterative operation, sequentially reading a plurality of sample data subsets from a sample data set, wherein each sample data subset comprises at least one piece of sample data;   entering each read sample data subset into a machine learning model; and   calculating gradient information of each of a plurality of parameters of the machine learning model, wherein after a set of gradient information of each parameter is calculated using one sample data subset, the sample data subset is deleted before a next sample data subset is read, and another set of gradient information of each parameter is calculated using the next sample data subset; and wherein the machine learning model has an initialized global parameter or is updated in a last iterative operation; and   in the process of the one iterative operation, accumulating a plurality of sets of calculated gradient information of each parameter to obtain an update gradient of each parameter, wherein the update gradient of each parameter is used to update the machine learning model.   
     
     
         8 . The data processing method according to  claim 7 , wherein the one iterative operation is iteratively performed in a plurality of iterative operations until the machine learning model converges or a specified quantity of iterations are completed in calculation, wherein
 in each of the plurality of iterative operations after the one iterative operation, after the machine learning model is updated using an update gradient obtained in an iterative operation, a next iterative operation is performed.   
     
     
         9 . The method according to  claim 7 , further comprising updating the machine learning model using the update gradient of each parameter during each iteration operation. 
     
     
         10 . The method according to  claim 8 , wherein accumulating, in each iterative operation, a plurality of sets of calculated gradient information of each parameter to obtain an update gradient of each parameter comprises:
 for the plurality of sets of gradient information of each parameter that are obtained based on the plurality of read sample data subsets, accumulating a plurality of sets of gradient information of a same parameter to obtain the update gradient of each parameter.   
     
     
         11 . The method according to  claim 8 , wherein accumulating, in each iterative operation, a plurality of sets of calculated gradient information of each parameter to obtain an update gradient of each parameter comprises:
 for one set of gradient information of each parameter obtained based on each sample data subset, accumulating one set of gradient information of a same parameter to obtain an accumulation gradient of each parameter, so that a plurality of accumulation gradients of each parameter are obtained based on the plurality of read sample data subsets, and   accumulating the plurality of accumulation gradients of each parameter to obtain the update gradient of each parameter.   
     
     
         12 . The method according to  claim 8 , wherein accumulating, in each iterative operation, a plurality of sets of calculated gradient information of each parameter to obtain an update gradient of each parameter comprises:
 for the plurality of sets of gradient information of each parameter that are obtained based on the plurality of read sample data subsets, collecting a plurality of sets of gradient information of a same parameter together, wherein the plurality of sets of gradient information of each parameter that are collected together are used as the update gradient of each parameter.   
     
     
         13 . The method according to  claim 8 , wherein in a process of entering each sample data subset into the machine learning model and calculating gradient information of each of the plurality of parameters of the machine learning model during each iterative operation, one piece of gradient information of each parameter is obtained correspondingly based on one piece of sample data in the sample data subset, wherein the sample data subset comprises at least one piece of sample data; and correspondingly, one set of gradient information of each parameter is obtained correspondingly based on one sample data subset, wherein the one set of gradient information comprises at least one piece of gradient information. 
     
     
         14 . The method according to  claim 8 , wherein updating the machine learning model using an update gradient of each parameter in each iterative operation comprises:
 updating the machine learning model according to a model update formula of stochastic gradient descent and using the update gradient of each parameter.   
     
     
         15 . The method according to  claim 8 , wherein the method runs on at least one computing node, and the computing node comprises at least one processor and a memory configured for the processor. 
     
     
         16 . The method according to  claim 8 , wherein in each iterative operation, the plurality of sample data subsets sequentially read from the sample data set are stored in the memory, and after gradient information of each parameter is calculated using one sample data subset, the sample data subset is deleted from the memory before a next sample data subset is read into the memory. 
     
     
         17 . The method according to  claim 16 , wherein a storage space occupied by one sample data subset is less than or equal to a storage space reserved for the sample data subset in the memory, and a storage space occupied by two sample data subsets is greater than the storage space reserved for the sample data subset in the memory. 
     
     
         18 . The method according to  claim 16 , further comprising:
 in each iterative operation, in a process of entering each sample data subset into the machine learning model and calculating gradient information of each of the plurality of parameters of the machine learning model, reading and using an intermediate calculation result stored in the memory, wherein the intermediate calculation result is used as input information to calculate the gradient information; and   after the intermediate calculation result is used, deleting the intermediate calculation result from the memory, wherein an operation of deleting the intermediate calculation result needs to be completed before a next sample data subset is read into the memory.   
     
     
         19 . A computing device, wherein the computing device comprises a mainboard, and a processor, a memory, a bus, and a communications interface that are connected using the mainboard, wherein the processor is connected to the memory via the bus; and when the computing device runs, the processor reads and executes an instruction in the memory or runs a hardware logic circuit of the processor, to make the computing device perform a data processing method, and reads a sample data subset into the memory via the communications interface;
 wherein the data processing method comprises one iterative operation, including:   sequentially reading a plurality of sample data subsets from a sample data set, wherein each sample data subset comprises at least one piece of sample data;   entering each read sample data subset into a machine learning model;   calculating gradient information of each of a plurality of parameters of the machine learning model, wherein after a set of gradient information of each parameter is calculated using one sample data subset, the sample data subset is deleted before a next sample data subset is read, and another set of gradient information of each parameter is calculated using the next sample data subset; and wherein the machine learning model has an initialized global parameter or is updated in a last iterative operation; and   accumulating a plurality of sets of calculated gradient information of each parameter to obtain an update gradient of each parameter, wherein the update gradient of each parameter is used to update the machine learning model.