Systems and Methods for Scheduling Neural Networks by Varying Batch Sizes

1. A system for scheduling at least one neural network for execution, comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions to cause the system to perform: determining a profile for one or more applications to be co-scheduled with the at least one neural network, the profile including a first set of hardware usage requirements for running the one or more corresponding applications; determining a batch size for the at least one neural network based on the determined profile for the one or more applications, the at least one neural network of the determined batch size having a second set of hardware usage requirements that complement the first set of hardware usage requirements such that a total of the first and second sets of hardware usage requirements is below a predetermined threshold; and scheduling the one or more applications and the at least one neural network based on the determined batch size.

2. The system of claim 1 , wherein determining the profile comprises executing one or more runtime tests on the one or more applications to determine one or more characteristics comprising the profile.

3. The system of claim 2 , wherein the one or more characteristics include one or more of maximum memory usage, total memory usage, memory usage over time, maximum processor usage, total processor usage, processor usage over time, maximum network usage, total network usage, or network usage over time.

4. The system of claim 1 , wherein determining the profile comprises: retrieving a profile from a database of profiles based on one or more characteristics of the one or more applications; or when no profile in the database matches the one or more applications: selecting a profile associated with one or more characteristics most similar to the one or more characteristics of the one or more applications.

5. The system of claim 4 , wherein selecting the profile associated with one or more characteristics most similar to the one or more characteristics of the one or more applications comprises at least one of: determining the profile associated with one or more characteristics with a minimal total distance to the one or more characteristics of the one or more applications; or determining the profile associated with one or more characteristics with minimal distances between each associated characteristic and a corresponding characteristic of the one or more characteristics of the one or more applications.

6. The system of claim 4 , wherein selecting the profile associated with one or more characteristics most similar to the one or more characteristics of the one or more applications comprises determining the profile having a feature vector based on one or more associated characteristics closest to a feature vector based on the one or more characteristics of the one or more applications.

7. The system of claim 6 , wherein the feature vector is determined using multiple dimensions.

8. The system of claim 7 , wherein the dimensions include one or more of maximum memory usage, total memory usage, memory usage over time, maximum processor usage, total processor usage, processor usage over time, maximum network usage, total network usage, or network usage over time.

9. The system of claim 6 , wherein the feature vector is determined using at least one neural network trained to compare feature vectors of applications.

10. The system of claim 1 , wherein determining the batch size comprises testing a plurality of batch sizes on the at least one neural network to determine the batch size that complements the profile for the one or more applications.

11. The system of claim 10 , wherein determining the batch size that complements the profile for the one or more applications comprises: determining the batch size that causes the at least one neural network to exhibit a profile having one or more associated characteristics at least a threshold distance from one or more characteristics of the profile for the one or more applications; or determining the batch size that causes the at least one neural network to exhibit a profile having one or more associated characteristics that result in minimal totals of each associated characteristic and a corresponding characteristic of the one or more characteristics of the profile for the one or more applications.

12. The system of claim 11 , wherein the one or more characteristics include one or more of maximum memory usage, total memory usage, memory usage over time, maximum processor usage, total processor usage, processor usage over time, maximum network usage, total network usage, or network usage over time.

13. The system of claim 1 , wherein determining the batch size comprises: retrieving a profile for the at least one neural network from a database of profiles based on one or more characteristics of the at least one neural network; or when no profile in the database matches the at least one neural network: selecting a profile associated with one or more characteristics most similar to the one or more characteristics of the at least one neural network.

14. The system of claim 13 , wherein the one or more characteristics include one or more of a number of hidden layers, a number of nodes per layer, or a connection scheme between nodes.

15. The system of claim 13 , wherein selecting the profile associated with one or more characteristics most similar to the one or more characteristics of the at least one neural network comprises at least one of: determining the profile associated with one or more characteristics with a minimal total distance to the one or more characteristics of the at least one neural network; or determining the profile associated with one or more characteristics with minimal distances between each associated characteristic and a corresponding characteristic of the one or more characteristics of the at least one neural network.

16. The system of claim 13 , wherein selecting the profile associated with one or more characteristics most similar to the one or more characteristics of the at least one neural network comprises determining the profile having a feature vector based on one or more associated characteristics closest to a feature vector based on the one or more characteristics of the at least one neural network.

17. The system of claim 16 , wherein the feature vector is determined using multiple dimensions.

18. The system of claim 16 , wherein the feature vector is determined using at least one neural network trained to compare feature vectors of neural networks.

19. A method for scheduling at least one neural network for execution, comprising: determining a profile for one or more applications co-scheduled with the at least one neural network, the profile including a first set of hardware usage requirements for running the one or more corresponding applications; determining a batch size for the at least one neural network based on the determined profile for the one or more applications, the at least one neural network of the determined batch size having a second set of hardware usage requirements that complement the first set of hardware usage requirements such that a total of the first and second sets of hardware usage requirements is below a predetermined threshold; and scheduling the one or more applications and the at least one neural network based on the determined batch size.

20. A non-transitory computer-readable storage medium storing a set of instructions that is executable by at least one processor to cause the processor to perform a method for scheduling a neural network for execution, the method comprising: determining a profile for one or more applications co-scheduled with at least one neural network, the profile including a first set of hardware usage requirements for running the one or more corresponding applications; determining a batch size for the at least one neural network based on the determined profile for the one or more applications, the at least one neural network of the determined batch size having a second set of hardware usage requirements that complement the first set of hardware usage requirements such that a total of the first and second sets of hardware usage requirements is below a predetermined threshold; and scheduling the one or more applications and the at least one neural network based on the batch size.