Coordinated Heterogeneous Processing of Training Data for Deep Neural Networks

1. A system comprising: a memory configured to store samples of training data for a Deep Neural Network (DNN); and a distributor configured to identify a plurality of work servers provisioned for training the DNN by processing the samples via models of the DNN, to receive information indicating Graphics Processing Unit (GPU) processing powers at the work servers, to determine differences in the GPU processing powers between the work servers based on the information, and to allocate the samples among the work servers based on the differences, wherein: the distributor is configured to determine throughputs of the work servers during training of the DNN, to dynamically select batch sizes for individual ones of the work servers between epochs during training of the DNN based on the throughputs, and to report the selected batch sizes to the work servers in order to control how samples are processed by the work servers; and each batch size defines a number of the samples for processing by one work server.

2. The system of claim 1 wherein: the processing of a batch is performed by a GPU cluster of a work server.

3. The system of claim 1 wherein: each of the work servers provides a gradient for updating parameters of the DNN.

4. The system of claim 2 further comprising: at least one modifier configured to receive input from the work servers upon completion of processing a number of samples equal to the batch sizes the work servers, to determine adjustments to the DNN based on the input, and to report the adjustments to the work servers for updating the models of the DNN.

5. The system of claim 1 wherein: the distributor is configured to determine an integer number of epochs of training to perform on the DNN; and the distributor is configured to allocate the samples among the work servers so that the integer number of the epochs will be completed but not exceeded during training.

6. The system of claim 1 wherein: at least one of the samples comprises an image.

7. The system of claim 1 wherein: at least one of the samples comprises a sound file.

8. A method comprising: identifying a plurality of work servers provisioned for training a Deep Neural Network (DNN) by processing samples of training data via models of the DNN; receiving information indicating Graphics Processing Unit (GPU) processing powers at the work servers; determining differences in the GPU processing powers between the work servers based on the information; allocating the samples among the work servers based on the differences; determining throughputs of the work servers during training of the DNN; dynamically selecting batch sizes for individual ones of the work servers between epochs during training of the DNN based on the throughputs; and reporting the selected batch sizes to the work servers in order to control how samples are processed by the work servers, wherein each batch size defines a number of the samples for processing by one work server.

9. The method of claim 8 further comprising: processing a batch via a GPU cluster of a work server.

10. The method of claim 8 wherein: each of the work servers provides a gradient for updating parameters of the DNN.

11. The method of claim 9 further comprising: receiving input from the work servers upon completion of processing a number of samples equal to the batch sizes the work servers; determining adjustments to the DNN based on the input; and reporting the adjustments to the work servers for updating the models of the DNN.

12. The method of claim 8 further comprising: determining an integer number of epochs of training to perform on the DNN; and allocating the samples among the work servers so that the integer number of epochs will be completed but not exceeded during training.

13. The method of claim 8 wherein: at least one of the samples comprises an image.

14. The method of claim 8 wherein: at least one of the samples comprises a sound file.

15. A non-transitory computer readable medium embodying programmed instructions which, when executed by a processor, are operable for performing a method comprising: identifying a plurality of work servers provisioned for training a Deep Neural Network (DNN) by processing samples of training data via models of the DNN; receiving information indicating Graphics Processing Unit (GPU) processing powers at the work servers; determining differences in the GPU processing powers between the work servers based on the information; allocating the samples among the work servers based on the differences; determining throughputs of the work servers during training of the DNN; dynamically selecting batch sizes for individual ones of the work servers between epochs during training of the DNN based on the throughputs; and reporting the selected batch sizes to the work servers in order to control how samples are processed by the work servers, wherein each batch size defines a number of the samples for processing by one work server.

16. The medium of claim 15 wherein the method further comprises: processing a batch via a GPU cluster of a work server.

17. The medium of claim 15 wherein: each of the work servers provides a gradient for updating parameters of the DNN.

18. The medium of claim 16 wherein the method further comprises: receiving input from the work servers upon completion of processing a number of samples equal to the batch sizes the work servers; determining adjustments to the DNN based on the input; and reporting the adjustments to the work servers for updating the models of the DNN.

19. The medium of claim 15 wherein the method further comprises: determining an integer number of epochs of training to perform on the DNN; and allocating the samples among the work servers so that the integer number of epochs will be completed but not exceeded during training.

20. The medium of claim 15 wherein: at least one of the samples comprises an image.