Scaling-Aware Traffic Request Prioritization for Scalable Connection-Based Resource

1. A system, comprising: one or more computers comprising respective processors and memory configured to implement a first request priority manager for a scalable connection-based resource, the first request priority manager configured to: receive request traffic for a scalable connection-based resource; monitor a connection capacity or request processing capacity of the scalable connection-based resource; in response to a determination that the scalable connection-based resource is at or within a threshold limit of a maximum capacity for the connection capacity or request processing capacity of the scalable connection-based resource, begin applying a first priority policy obtained from a policy data store to newly received request traffic for the scalable connection-based resource to determine a priority of some of the newly received requests over others; and in response to a determination that the scalable connection-based resource is no longer at or within the threshold limit of the maximum capacity for the scalable connection-based resource, cease applying the first priority policy to newly received request traffic for the scalable connection-based resource.

2. The system of claim 1, wherein the first request priority manager comprises a connection pooler configured to: establish respective front-end connections with a plurality of requesters for receiving the request traffic for the scalable connection-based resource; and maintain a pool of backend connections to the scalable connection-based resource for sending the request traffic to the scalable connection-based resource; wherein to apply the first priority policy the first request priority manager is requests received via the respective front-end connections are sent over the backend connections to the scalable connection-based resource.

3. The system of claim 2, wherein, responsive to an increase in the request traffic for the scalable connection-based resource, the connection pooler is configured to request one or more additional backend connections to the scalable connection-based resource to increase a number of connections in the pool of backend connections, wherein the determination that the scalable connection-based resource is at or within the threshold limit of the maximum capacity is based on the ability of the scalable connection-based resource to accept the one or more additional backend connections.

4. The system of claim 3, wherein the scalable connection-based resource comprises a group of resources instances configured to establish the backend connections with the connection pooler and process requests from the request traffic received via respective connections of the backend connections, wherein the scalable connection-based resource is scalable to increase or decrease a number of resources instances in the group of resources instances; and wherein the first request priority manager is configured to request the scalable connection-based resource to increase the number of resources instances in the group of resources instances in response to an indication that the scalable connection-based resource is unable to accept the one or more additional backend connections.

5. The system of claim 2, further comprising: one or more computers comprising respective processors and memory configured to implement a second request priority manager, implemented at the scalable connection-based resource and configured to: receive request traffic for the scalable connection-based resource over the backend connections; monitor the connection capacity or request processing capacity of the scalable connection-based resource; and in response to a determination that the scalable connection-based resource is at or within a threshold limit of a maximum capacity for the scalable connection-based resource, begin applying a second priority policy to newly received request traffic for the scalable connection-based resource to prioritize some requests over others, wherein the second priority policy is different from the first priority policy.

6. A method, performed by one or more processors of one or more computing devices, the method comprising: receiving, by a first request priority manager, request traffic for a scalable connection-based resource; monitoring a connection capacity or request processing capacity of the scalable connection-based resource; applying, by the first request priority manager and in response to a determination that the scalable connection-based resource is at or within a threshold limit of a maximum capacity for the connection capacity or request processing capacity of the scalable connection-based resource, a first priority policy obtained from a policy data store to newly received request traffic for the scalable connection-based resource to determine a priority of some of the newly received requests over others; and in response to determining that the scalable connection-based resource is no longer at or within the threshold limit of the maximum capacity for the scalable connection-based resource, cease applying the first priority policy to newly received request traffic for the scalable connection-based resource.

7. The method of claim 6, wherein the first request priority manager comprises a connection pooler configured to: establishing, by a connection pooler of the first request priority manager, respective front-end connections with a plurality of requesters for receiving the request traffic for the scalable connection-based resource; and maintaining, by the connection pooler, a pool of backend connections to the scalable connection-based resource for sending the request traffic to the scalable connection-based resource; wherein said applying the first priority policy comprises determining, according to the first priority policy, which requests received via the respective front-end connections are sent over the backend connections to the scalable connection-based resource.

8. The method of claim 7, further comprising: responsive to an increase in the request traffic for the scalable connection-based resource, requesting by the connection pooler one or more additional backend connections to the scalable connection-based resource to increase a number of connections in the pool of backend connections, wherein said determining that the scalable connection-based resource is at or within the threshold limit of the maximum capacity is based on the ability of the scalable connection-based resource to accept the one or more additional backend connections.

9. The method of claim 8, wherein: the connection pooler comprises a group of connection pool instances for maintaining respective connections of the pool of backend connections to the scalable connection-based resource; said increase the number of connections in the pool of backend connections comprises increase a number of connection pool instances in the group of connection pool instances; and the method further comprises: begin applying, by the first request priority manager, the first priority policy to the request traffic responsive to reaching a maximum number of connection pool instances or a maximum number of connections in the pool of backend connections.

10. The method of claim 6, wherein: the first priority policy specifies one or more data objects stores at the scalable connection-based resource; and said applying the first priority policy comprises prioritizing requests that access at least one of the one or more data objects specified in the first priority policy.

11. The method of claim 6, wherein: the first priority policy specifies one or more requestor identifiers (IDs); and said applying the first priority policy comprises prioritizing respective requests that indicate one of the one or more requestor IDs specified in the first priority policy as a source of the respective request.

12. The method of claim 6, further comprising: providing, by the first request priority manager, an interface for a user of the scalable connection-based resource to specify the first priority policy.

13. The method of claim 6, wherein: the first request priority manager comprises an artificial intelligence component; and the method further comprises specifying or modifying, by the artificial intelligence component, the first priority policy based on historical request traffic to the scalable connection-based resource.

14. One or more non-transitory computer-readable media storing program instructions executable on or across one or more processors implement a first request priority manager to: monitor a connection capacity or request processing capacity of a scalable connection-based resource to which request traffic is directed; begin application, in response to a determination that the scalable connection-based resource is at or within a threshold limit of a maximum capacity for the connection capacity or request processing capacity of the scalable connection-based resource, of a first priority policy obtained from a policy data store to newly received request traffic for the scalable connection-based resource to determine a priority of some of the newly received requests over others; and cease application, in response to a determination that the scalable connection-based resource is no longer at or within the threshold limit of the maximum capacity for the scalable connection-based resource, of the first priority policy to newly received request traffic for the scalable connection-based resource.

15. The one or more non-transitory computer-readable media of claim 14, storing program instructions executable to implement a connection pooler at the first request priority manager, the connection pooler to: establish respective front-end connections with a plurality of requesters for receiving the request traffic for the scalable connection-based resource; and maintain a pool of backend connections to the scalable connection-based resource for sending the request traffic to the scalable connection-based resource; wherein said application of the first priority policy comprises determine, according to the first priority policy, which requests received via the respective front-end connections are sent over the backend connections to the scalable connection-based resource, and which requests received via the respective front-end connections are not sent over the backend connections to the scalable connection-based resource.

16. The one or more non-transitory computer-readable media of claim 15, storing program instructions executable to cause the connection pooler to: responsive to an increase in the request traffic for the scalable connection-based resource, request one or more additional backend connections to the scalable connection-based resource to increase a number of connections in the pool of backend connections; and base the determination that the scalable connection-based resource is at or within the threshold limit of the maximum capacity on the ability of the scalable connection-based resource to accept the one or more additional backend connections.

17. The one or more non-transitory computer-readable media of claim 15, wherein: the connection pooler comprises a group of connection pool instances for maintaining respective connections of the pool of backend connections to the scalable connection-based resource; to increase the number of connections in the pool of backend connections, the program instructions are executable to increase a number of connection pool instances in a group of connection pool instances; and the program instructions are executable to perform said application, by the first request priority manager, of the first priority policy to the request traffic responsive to reaching a maximum number of connection pool instances or a maximum number of connections in the pool of backend connections.

18. The one or more non-transitory computer-readable media of claim 15, wherein: the first priority policy specifies one or more data objects stores at the scalable connection-based resource; and to perform said application of the first priority policy the program instructions are executable to prioritize requests that access at least one of the one or more data objects specified in the first priority policy.

19. The one or more non-transitory computer-readable media of claim 15, wherein: the first priority policy specifies one or more requestor identifiers (IDs); and to perform said application of the first priority policy the program instructions are executable to prioritize respective requests that indicate one of the one or more requestor IDs specified in the first priority policy as a source of the respective request.

20. The one or more non-transitory computer-readable media of claim 15, wherein the program instructions are executable to cause the first request priority manager to determine an anomalous pattern in the request traffic and modify the first priority policy to change a prioritization for new request traffic corresponding to the determined anomalous pattern.