Dynamic network bandwidth in distributed deep learning training

Abstract

Embodiments of a method are disclosed. The method includes performing distributed deep learning training on a batch of training data. The method also includes determining training times representing an amount of time between a beginning batch time and an end batch time. Further, the method includes modifying a communication aspect of the communication straggler to reduce a future network communication time for the communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to the centralized parameter server determining that the learner is the communication straggler.

What is claimed is: 
     
       1. A computer-implemented method, comprising:
 performing distributed deep learning training on a batch of training data; 
 determining a training time representing an amount of time between:
 a beginning batch time for a learner; and 
 an end batch time for the learner; 
 
 determining that the learner is a communication straggler by determining that the training time exceeds a predetermined threshold time; and 
 modifying a communication aspect of the learner to reduce a future network communication time for the communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to a centralized parameter server determining that the learner is the communication straggler, wherein modifying the communication aspect comprises compressing the future result before sending the future result to the centralized parameter server, wherein the future result is compressed using a compression rate based on a network communication time of the communication straggler. 
 
     
     
       2. The method of  claim 1 , further comprising:
 determining a plurality of network communication times representing an amount of time between:
 a plurality of batch end times; and 
 a plurality of times when the centralized parameter server receives a plurality of results; and 
 
 identifying the communication straggler based on the plurality of network communication times and a threshold network communication time. 
 
     
     
       3. The method of  claim 1 , further comprising modifying a processing aspect of the communication straggler, wherein modifying the processing aspect comprises:
 determining a size of the new batch of training data; and 
 distributing a reduced amount of communication straggler training data to a plurality of remaining learners for performing the distributed deep learning training. 
 
     
     
       4. The method of  claim 3 , further comprising:
 performing the distributed deep learning training on the new batch of training data by the communication straggler; and 
 performing the distributed deep learning training on the reduced amount of communication straggler training data by the plurality of remaining learners. 
 
     
     
       5. The method of  claim 1 , further comprising modifying a processing aspect of the communication straggler, wherein modifying the processing aspect comprises increasing a frequency rate of a computational processor of the communication straggler. 
     
     
       6. The method of  claim 1 , further comprising:
 performing the distributed deep learning training on a plurality of batches of training data using a plurality of learners; 
 determining a plurality of training times representing an amount of time between:
 when the plurality of learners begin generating a corresponding plurality of results for the centralized parameter server for the distributed deep learning training; and 
 when the plurality of learners send the plurality of results to the centralized parameter server; and 
 
 determining a plurality of network communication times representing an amount of time between:
 when the plurality of learners send the plurality of results to the centralized parameter server for the distributed deep learning training; and 
 when the centralized parameter server receives the plurality of results. 
 
 
     
     
       7. A computer program product comprising program instructions stored on a computer readable storage medium, the program instructions executable by a processor to cause the processor to perform a method comprising:
 a learner performing distributed deep learning training on a batch of training data; 
 the learner determining a plurality of training times representing an amount of time between:
 a beginning batch time; and 
 an end batch time; 
 
 the learner determining that the learner is a communication straggler by determining that a training time of the learner exceeds a predetermined threshold time; and 
 the learner modifying a communication aspect of the communication straggler to reduce a future network communication time for the communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to a centralized parameter server determining that the learner is the communication straggler, wherein modifying the communication aspect comprises compressing the future result before sending the future result to the centralized parameter server, wherein the future result is compressed using a compression rate based on a network communication time of the communication straggler. 
 
     
     
       8. The computer program product of  claim 7 , further comprising:
 determining a plurality of network communication times representing an amount of time between:
 a plurality of batch end times; and 
 when the centralized parameter server receives a plurality of results; and 
 
 identifying the communication straggler based on the plurality of network communication times and a threshold network communication time. 
 
     
     
       9. The computer program product of  claim 7 , further comprising modifying a processing aspect of the communication straggler, wherein modifying the processing aspect comprises:
 determining a size of the new batch of training data; and 
 distributing a reduced amount of communication straggler training data to a plurality of remaining learners for performing the distributed deep learning training. 
 
     
     
       10. The computer program product of  claim 9 , further comprising:
 performing the distributed deep learning training on the new batch of training data by the communication straggler; and 
 performing the distributed deep learning training on the reduced amount of communication straggler training data by the plurality of remaining learners. 
 
     
     
       11. The computer program product of  claim 7 , further comprising modifying a processing aspect of the communication straggler, wherein modifying the processing aspect comprises increasing a frequency rate of a computational processor of the communication straggler. 
     
     
       12. The computer program product of  claim 7 , further comprising:
 performing the distributed deep learning training on a plurality of batches of training data using a plurality of learners; 
 determining the plurality of training times representing an amount of time between:
 when the plurality of learners begins generating a corresponding plurality of results for the centralized parameter server for the distributed deep learning training; and 
 when the plurality of learners sends the plurality of results to the centralized parameter server; and 
 
 determining a plurality of network communication times representing an amount of time between:
 when a plurality of learners sends the plurality of results to the centralized parameter server for the distributed deep learning training; and 
 when the centralized parameter server receives the plurality of results. 
 
 
     
     
       13. A system comprising:
 a computer processing circuit; and 
 a computer-readable storage medium storing instructions, which, when executed by the computer processing circuit, are configured to cause the computer processing circuit to perform a method comprising: 
 performing distributed deep learning training on a batch of training data; 
 determining a training time representing an amount of time between:
 a beginning batch time for a learner; and 
 an end batch time for the learner; 
 
 determining that the learner is a communication straggler by determining that the training time exceeds a predetermined threshold time; and 
 modifying a communication aspect of the learner to reduce a future network communication time for the communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to a centralized parameter server determining that the learner is the communication straggler, wherein modifying the communication aspect comprises compressing the future result before sending the future result to the centralized parameter server, wherein the future result is compressed using a compression rate based on a network communication time of the communication straggler. 
 
     
     
       14. The system of  claim 13 , the method further comprising:
 determining a plurality of network communication times representing an amount of time between:
 a plurality of batch end times; and 
 a plurality of times when the centralized parameter server receives a plurality of results; and 
 
 identifying the communication straggler based on the plurality of network communication times and a threshold network communication time.