Using Distributed Timers in an Overlay Network

1. At a node in an overlay network, the node including a processor and system memory, the overlay network including a plurality nodes, each node in the plurality of nodes being assigned responsibility for a range of identifiers on the overlay network, a method for replicating timers within the overlay network, the method comprising: receiving data for a process at the node, the node being assigned responsibility responsible for a specified range of identifiers on the overlay network, the process identified by an identifier within the specified range of identifiers; the process processing the data to partially completed a portion of work; the process outputting a timer based on the partially completed portion of work, the timer indicative of an amount of time that further progress towards completing the portion of work is to be suspended, the timer including deferred data that is to be processed when further progress towards completing the portion of work is resumed and the timer including a designated time indicating when the timer is to expires and further progress towards completing the portion of work is to be resumed; supplementing the timer with the identifier; and replicating the timer to a plurality of other nodes on the overlay network for storage in durable storage at the plurality of other nodes, replicating the timer increasing the availability of the deferred data such that if responsibility for the identifier is subsequently reassigned to one of the plurality of other nodes, the deferred data is available to the process when the timer expires and further progress towards completing the portion of work is to resume.

2. The method as recited in claim 1 , wherein receiving data for a process at the node comprises receiving data for a workflow having an identifier within the specified range of identifiers.

3. The method as recited in claim 1 , wherein receiving data for a process at the node comprises receiving data from a front end node, the front end node providing an interface between the overlay network and systems external to the overlay network.

4. The method as recited in claim 1 , wherein the process processing the data to perform some work comprises performing a portion of workflow.

5. The method as recited in claim 1 , wherein the process processing the data to perform some work comprises queuing the data within a workflow process.

6. The method as recited in claim 5 , further comprising: tracking the timer in system memory; determining that the designated time has been reached; accessing the timer from the durable storage; determining from the timer that the identifier is still in within the specified range of identifiers; and sending the data to the process in response to determining that the designated time has been reached and in response to determining that the identifier is still in within the specified range of identifiers.

7. The method as recited in claim 1 , further comprising: storing the timer, including the data, the designated time, and the identifier, in durable storage.

8. The method as recited in claim 1 , wherein the overlay network is a ring overlay network and wherein the node is a node on the ring overlay network.

9. At a node in an overlay network, the node including a processor and system memory, the overlay network including a plurality nodes, each node in the plurality of nodes assigned responsible for a range of identifiers on the overlay network, the node assigned responsibility for a specified range of identifiers on the overlay network, a method for assuming ownership of a replicated timer, the method comprising: receiving a replicated timer related to a process at another node on the overlay network, the replicated timer representing that the process has partially completed a portion of work at the other node, the timer indicating that further progress towards completing the portion of work is suspended for some period of time, the timer including timer data that is to be processed when further progress toward completing the portion of work resumes, a designated time when the timer is to expire, and an identifier for the process, the identifier being outside the specified range of identifiers assigned to the node; detecting a change in the node configuration on the overlay network subsequent to receiving the replicated timer; updating the specified range of identifiers for the node based on the detected change in node configuration, the update to the specified range of identifiers changing the assigned responsibilities for the node; determining that the identifier for the process is within the updated specified range of identifiers such that the node has been assigned responsibility for when the timer expires, in view of the changed node configuration; determining that the designated time has been reached subsequent to the node being assigned responsibility for indicating when the timer expires; indicating to the process that that timer has expired; and submitting the timer data to the process in response to the indication such that process can use the timer data to progress further towards completing the portion of work.

10. The method as recited in claim 9 , wherein receiving a replicated timer for a process at another node comprises receiving a replicated timer for a workflow process at another node.

11. The method as recited in claim 9 , wherein detecting a change in the node configuration on the overlay network comprises detecting that a node has departed the overlay network.

12. The method as recited in claim 9 , wherein detecting a change in the node configuration on the overlay network comprises detecting that a node has joined the overlay network.

13. The method as recited in claim 9 , wherein updating the specified range of identifiers for the node comprises the node increasing the specified range of identifiers to include more identifiers.

14. The method as recited in claim 9 , further comprising tracking the timer in system memory.

15. The method as recited in claim 9 , further comprising locally activating the process at the node.

16. The method as recited in claim 15 , wherein submitting the timer data to the process comprises queuing the data within a workflow process for processing by the workflow instance.

17. The method as recited in claim 9 , wherein the overlay network is a ring overlay network and wherein the node is a node on the ring overlay network.

18. An overlay ring network, the overlay ring network including a plurality of peer nodes, each peer node assigned responsibility for a range of identifiers within the distributed hash table, each node including: one or more processors; and system memory having stored thereon computer executable instructions that, when executed at one of the one or more processors, cause the node to participate in workflow processing within the overlay network, including each node being configured to a) initiate a process for a workflow, b) progress through a workflow as the node assigned responsibility for the workflow, and c) transition to the node assigned responsibility for a workflow, wherein a) initiating a process for a workflow includes: locally activating a workflow process for a workflow at the node; and co-locating a queue and a workflow runtime for the workflow within a locally activated process; b) progressing through a workflow as the node assigned responsibility for the workflow includes: receiving data for the workflow at the node, the workflow identified by an identifier within the specified range of identifiers; processing the received data at the workflow runtime to perform some work; outputting a timer based on the performed work, the timer indicative of an amount of time the workflow is to sleep before again awakening to process data, the timer including deferred data that is to be processed when the workflow awakens and the timer including a designated time when the workflow is to be awaken; replicating the timer to a plurality of other nodes on the ring overlay network, replicating the timer increasing the availability of the timer such that if responsibility for the workflow is subsequently reassigned to one of the plurality of other nodes, the timer is available to a workflow runtime at the reassigned node; and c) transitioning to the node assigned responsibility for a workflow includes: receiving a replicated timer for a workflow another node on the overlay network, the timer representing that a workflow the timer indicating the process is to sleep for some period of time before awakening to process data, the replicated timer including timer data that is to be processed when the process awakens, a designated time when the process is to be awaken, and an identifier for the process, the identifier being outside the specified range of identifiers assigned to the node; detecting a change in the node configuration on the overlay network subsequent to receiving the replicated timer; updating the specified range of identifiers for the node based on the detected change in node configuration, the update to the specified range of identifiers changing the assigned responsibilities for the node; determining that the identifier for the process is within the updated specified range of identifiers such that the node has been assigned responsibility for the process in view of the changed node configuration; determining that the designated time has been reached subsequent to the node being assigned responsibility for the process; locally activating a workflow instance for the workflow at the node in response to determining that the designated time has been reached; submitting the timer data to a queue for the workflow instance; and the workflow instance at the node processing the timer data to continue the portion of work from the point of partial completion reached at the other node.

19. The system as recited in claim 18 , wherein the overlay network is based on a distributed hash table.

20. The system as recited in claim 18 , wherein nodes store timers in durable storage.