Failover handling for pods executing an application in a high availability mode

Container orchestration platforms, such as Kubernetes®, manage deployment of containerized applications in what are commonly referred to as pods. Each pod may include one or more containers for executing application processes therein. Kubernetes is at least one platform that does not natively include features that support high availability for an application executing in pods. For instance, an application that supports high availability may designate a set of active pods and a set of standby pods. If one of the active pods fails, then the application will designate one of the standby pods to take over for the failed pod. However, since Kubernetes does not support the application's failover capabilities, a service object that avails one or more of the pods to a network will still point to the failed pod once that pod is respawned within the standby set as directed by the application. The service will not point to the pod that the application activated to take over for the failed pod.

The technology disclosed herein enables a service manager of a container orchestration platform to handle failovers of pods executing an application in a high availability mode. In a particular example, a method includes receiving pod information about the pods. The pod information includes unique application identifiers generated by the application for the pods and indications of which of the pods are active and which of the pods are on standby. The pods are orchestrated by the container orchestration platform. The method further includes configuring service objects provided by the container orchestration platform to each correspond to respective ones of the pods based on the unique application identifiers. The service objects expose corresponding ones of the pods to a network. The method also includes receiving updated pod information indicating a first pod of the pods, which was previously on standby, is now active with a first application identifier of the unique application identifiers previously assigned to a second pod of the pods that failed. Additionally, the method includes reconfiguring a service object of the service objects associated with the first application identifier to correspond to the first pod instead of the second pod.

In other examples, an apparatus performs the above-recited method and program instructions stored on computer readable storage media direct a processing system to perform the above-recited method.

Container orchestration platforms that organize containers into pods may use service objects (often simply referred to as services) to avail those pods to a communication network. When data on a network is intended for an application executing in a pod, a service object implemented by the container orchestration platform and corresponding to the pod directs the data to the pod. Kubernetes is an example container orchestration platform that uses service objects but other container orchestration platforms may use service objects, or something analogous thereto, in a similar manner. Traditionally, when a pod fails, a service object will continue to associate with that pod. As such, communications with the pod and processing by the pod will stop until the pod has recovered (e.g., is respawned by the container orchestration platform). Even if the pod takes a relatively small amount of time to recover, the time may still be enough to have adverse consequences for functionality being provided by an application executing in the pod. For example, if the application provides transcoding functionality for real-time user communication sessions, then failure of the pod may cause an undesirable gap in communications or may cause the sessions to drop altogether.

An application may be configured to operate in a high availability mode across the pods to avoid issues caused by a pod's failure. When an active pod fails, another pod waiting on standby can be activated by the application to take the place of the failed pod. However, the service object for the failed pod will not automatically switch its association from the failed pod to the newly activated pod, which prevents the application in the newly activated pod from taking over for the application in the failed pod. The service managers described in the examples below reconfigure a service object when a failure occurs in a pod associated therewith. The service manager identifies the newly activated pod and associates the service object with the newly activated pod instead of the failed pod. Communications that were to be exchanged with the failed pod can then be exchanged with the newly activated pod in place of the failed pod.

illustrates implementationfor handling failover of pods executing an application in a high availability mode. Implementationincludes service manager, active pods, standby pods, active services, standby services, servers, and communication network. Active pods, standby pods, active services, and standby servicesimplemented by container orchestration platformand execute on one or more servers(although, other types of computing systems may be used). Communication networklinks serverstogether and may connect serverswith other systems and networks (e.g., the Internet). Service manageris an application process executed by serversfor managing active servicesand standby services. Service managermay be implemented in a pod executing on servers. In some examples, service managermay be a process executing on a computing system other than serversand in communication with serversover communication network.

In operation, applicationis an application that supports operating in a high availability mode within pods orchestrated by container orchestration platform. Applicationmay perform any type of processing function. In this example, applicationis executing within one or more active podsand standby pods. The instances of applicationthat are executing in active podsare active and ready to perform as configured (e.g., process incoming communications) if not actively performing already. The instances of applicationexecuting in standby podsare on standby and will not perform actively until activated (e.g., when a one of active podsfails). Active servicesare service objects that expose active pods(i.e., make active podsaccessible) to a network over which active podscan exchange communications. The network may be a logical network connecting pods. Similarly, service objectare service objects that expose standby podsto the network. Although, since standby podsare on standby, little to no such communications are typically exchanged with standby pods.

While a single service object in other examples may service multiple pods, the service objects in this example have a one-to-one correspondence with their associated pods. For example, service objectis one of active servicesand is associated with podof active pods. A different service object of active serviceswould then be associated with another pod of active pods. Likewise, service objectis one of standby servicesand is associated with podof standby pods. A different service object of standby serviceswould then be associated with another pod of standby pods. Service managermaintains this one-to-one correspondence to ensure one of active servicesis connected to one of active podseven when one of active podsfails.

illustrates operationto handle failover of pods executing an application in a high availability mode. In operation, service managerreceives pod information about pods-(). The pod information includes unique application identifiers generated by applicationfor pods-and indications of which of the pods are active (i.e., included in active pods) and which of the pods are on standby (i.e., included in standby pods). The unique application identifiers can be any data (e.g., character, string of characters, numbers, etc.) that is unique at least among active pods, although, the unique application identifiers may be unique among all of pods-. In this example, for simplicity the unique application identifiers are single digit numbers. For instance, 1 is the unique application identifier for podand 2 is the unique application identifier for pod. The unique application identifiers may be assigned by application(e.g., applicationon each pod may assign its own unique application identifier after ensuring the uniqueness of the unique application identifier) or may be assigned by an application controller for application. Pods-are identified to service managerby their respective labels. Like the unique application identifiers, a label can be any data that uniquely identifies a particular pod from amongst others of pods-. For clarity, the pod labels in this example closely relate to the reference numbers used in implementationfor pods-(e.g., the label for podis POD-).

The received pod information, therefore, informs the service managerabout which pods, identified by their unique labels, are active or on standby and which unique application identifier is assigned to the applicationthereon. For instance, the pod information indicates that pod label POD-is associated with an active pod having unique application identifier 1 and pod label POD-is associated with a standby pod having unique application identifier 2. The same type of information is received for the rest of pods-. In this example, each of pods-sends service managertheir respective portion pod information. In other examples, an application controller for applicationmay collect the pod information from pods-and pass the pod information along to service manager. The pod information may be pushed to service manageror may be sent in response to a request from service manager.

Service managerconfigures service objects-provided by container orchestration platformto each correspond to respective ones of pods-based on the unique application identifiers (). If one or more of service objects-do not already exist, then configuring service objects-includes creating those service objects that do not already exist. Each of service objects-is configured to correspond to one of pods-. In this example, each of service objects-includes a label selector to select a pod which the service object is exposing to the network. Label selectors may be configured with multiple labels to enable the label selector to select from multiple pods. For example, the label selector of a service object may include three labels of pods that can be selected. The label selector can then select one of the three pods to handle incoming network traffic, which may be useful to load balance across the pods. In this case, each of service objects-corresponds to a single one of pods-rather than multiple pods so the label selectors of service objects-are only given one pod label from which to select a pod. For instance, service managerconfigures the label selector of service objectto select label POD-and the label selector of service objectto select label POD-. Service managerfurther associates each of service objects-with the unique application identifier of the pod having the label the label selector is configured to select. For example, service objectis associated with unique application identifier 1 because podis assigned unique application identifier 1 and service objectis associated with unique application identifier 2 because podis associated with unique application identifier 2. Service managermay track the associations internally (e.g., in a data structure) or may configure service objects-to indicate the association (e.g., tag the service objects-with their respective unique application identifiers).

After configuring service objects-based on the pod information, service managerreceives updated pod information (). The updated pod information in this example indicates pod, which was previously on standby, is now active and associated with unique application identifier 1, which was previously assigned to pod. Podis now active because podfailed and applicationactivated podto take the place of podin active pods. When activating pod, applicationassigns the unique application identifier of failed podto the newly activated podso that applicationon podcan take over the tasks for applicationon pod. Instances of applicationon the remaining pods may negotiate among themselves to determine which of standby podsshould be activated upon determining podfailed or a controlling instance of applicationmay determine which of standby podsshould be activated. Like the pod information from above, the updated pod information may be received directly from podor may be received from an application controller.

In response to being notified in the updated pod information that podis now active with Unique Application Identifier 1, service managerreconfigures service objectto correspond to podinstead of pod(). To reconfigure service object, service managermay first identify the service object associated with unique application identifier 1 from within active services, which is service objectin this case. Service managermay then change the label selector of service objectto select the label for pod, which is POD-. After reconfiguring, any network traffic that service objectwould have directed to podwould instead be directed to pod, as service objectis still associated with the pod having unique application identifier 1.

Service object, which was associated with podwhen podwas on standby, may not be reconfigured by service manager, at least not immediately. Since service objectis one of standby services, it does not matter that service objectstill has a label selector configured to select podbecause standby servicesdo not actually handle traffic due to their standby role. Once podrecovers from its failure, service managermay receive updated pod information notifying service managerthat podhas recovered. For example, the updated pod information may indicate that the pod having label POD-is on standby with the unique application identifier 2. Service managermay then identify service objectas being the one of standby servicesassociated with unique application identifier 2 and configure the label selector of service objectto select pod label POD-.

illustrates operational scenariofor handling failover of pods executing an application in a high availability mode. In operational scenario, applicationis executing in pods-and supports high availability. Applicationis managed by application controllerwhich is executing in pod. Application controllerregulates the number of pods that are executing application, which of those pods are active versus standby, and handles pod information updates to service manager. In other examples, rather than being a distance process, one instance of applicationmay be designated a master and perform similar functions to that described for application controller. Like application controller, service managerexecutes in podand manages service objects-for respective ones of pods-. While not shown, pods-and service objects-are executing on a container orchestration platform like container orchestration platformand on computing systems like servers.

During operational scenario, application controllerdetermines that applicationwill execute with three active application pods and two standby application pods. The number of active versus standby pods may be defined by a user, coded into application controller, requested by application, or may be indicated from some other source. In some examples, the number of active pods for applicationmay be dependent on demand for applicationand the number of standby pods may be determined as a ratio of the active pods. With respect to this example, the ratio may indicate that the number of standby pods should be 40% of the number of active pods, which leads to the 3-2 ratio of active to standby pods. Accordingly, application controllerdirects the container orchestration platform to create pods-with applicationtherein and assigns unique application identifiers and roles at step 1 to each of pods-. Like in the example above, the unique application identifier for each of pods-is a single digit number but can be any data that is unique among the others of pods-. In this example, application controllerdesignates pods-to be active and pods-to be on standby. Pods-are also shown to have pod labels related to their reference numbers in operational scenario, although, other unique identifiers may be used for the labels. The pod labels may be automatically assigned by the container orchestration platform, may be assigned by application controller, or may be assigned by some other source.

Application controlleralso sends the unique application identifiers with their associated pod labels and roles at step 2 to service manager. Based on the information received from application controller, service manageris aware of which pods, as identified by their labels, are active/standby and their respective unique application identifiers. In response to the information, service managerlabels and configures service objects-to correspond to respective ones of pods-at step 3. The labels for service objects-can be any data but, in this example, service manageruses the labels to indicate information about the pods with which each of service objects-is associated. For instance, the label selector of service objectis configured to select label POD-to select pod. Podis active and has unique application identifier 1 so service managerlabels service objectwith the label Active-svc-1. Similarly, service objectis labelled Standby-svs-4 because the label selector for service objectis configured to select label POD-for pod, which is a standby pod. The remaining ones of service objects-are also labelled and configured similarly based on their corresponding ones of pods-. When handling network traffic each of service objects-, as configured, will select a single corresponding one of pods-at step 4.

illustrates additional steps of operational scenariofor handling failover of pods executing an application in a high availability mode. Podfails in this example. In response to determining that podfailed, application controllerreassigns unique application identifier 2, which was assigned to pod, to podand changes the role of podto active at step 5. Application controllermay have determined podfailed because podstopped responding to messages from application controller, application controllerstopped receiving heartbeat messages from pod, or based on some other indication that podhas failed. Since simply changing podto active will not enable podto take over for pod, application controllersends updated information at step 6 to service manager. The updated information indicates that the pod with label POD-(i.e., pod) is now associated with unique application identifier 2. The updated information may explicitly indicate that podis no longer associated with unique application identifier 2 and/or has failed or service managermay infer from the updated information that podis no longer active with unique application identifier 2.

In response to receiving the updated information, service manageridentifies service objectas being the service object that corresponds to unique application identifier 2 and updates the label selector of service objectat step 7 to select the label POD-instead of POD-, which was previously the pod label associated with unique application identifier 2. The label of service objectitself need not be changed because it is still associated with an active pod having unique application identifier 2. Updating the label selector in service objectcauses service objectto select podto handle network traffic at step 8 rather than selecting podas previously would have occurred.

illustrates further steps of operational scenariofor handling failover of pods executing an application in a high availability mode. Podrecovers from the failure described above (e.g., is respawned by the container orchestration platform). Upon application controllerrecognizing the podhas recovered, application controllerreassigns unique application identifier 4 to podand places podon standby at step 9. Unique application identifier 4 was previously the unique application identifier assigned to podprior to podbeing activated. Application controllerassigns unique application identifier 4 rather than a different unique application identifier because a service object (i.e., service object) is already associated with unique application identifier 4 from when podwas assigned unique application identifier 4.

Application controllersends updated information at step 10 to service manager. The updated information indicates that the pod with label POD-(i.e., pod) is now associated with unique application identifier 4. In response to receiving the updated information, service manageridentifies service objectas being the service object that corresponds to unique application identifier 4 and updates the label selector of service objectat step 11 to select the label POD-instead of POD-, which was previously the pod label associated with unique application identifier 4. The label of service objectitself need not be changed because it is still associated with a standby pod having unique application identifier 4. Updating the label selector in service objectcauses service objectto select podto handle network traffic at step 12 rather than selecting podas previously would have occurred. Granted, since podis a standby pod, no traffic will likely be exchanged with podvia service object.

illustrates a computing systemthat is representative of any system or collection of systems in which the various processes, systems, programs, services, and scenarios disclosed herein may be implemented. Examples of computing systeminclude, but are not limited to, desktop computers, laptop computers, server computers, routers, web servers, cloud computing platforms, and data center equipment, distributed computing systems, as well as any other type of physical or virtual server machine, physical or virtual router, container, and any variation or combination thereof. It should be understood virtualized components are executing on physical hardware like that of computing system.

Computing systemmay be implemented as a single apparatus, system, or device or may be implemented in a distributed manner as multiple apparatuses, systems, or devices. Computing systemmay include, but is not limited to, processing system, storage system, software, communication interface system, and user interface system. Processing systemmay be operatively coupled with storage system, communication interface system, and user interface system.

Processing systemmay load and execute softwarefrom storage system. Softwaremay include and implement process, which may be representative of any of the operations for handling failover of pods executing an application in a high availability mode (e.g., operation). Accordingly, computing systemmay be service manageror may be a host system for a virtualized example of service manager(e.g., service manager). Computing systemmay also represent physical servers—including combinations thereof. When executed by processing system, softwaremay direct processing systemto operate as described herein for at least the various processes, operational scenarios, and sequences discussed in the foregoing implementations. Computing systemmay optionally include additional devices, features, or functionality not discussed for purposes of brevity.

In some embodiments, processing systemmay comprise a micro-processor and other circuitry that retrieves and executes softwarefrom storage system. Processing systemmay be implemented within a single processing device but may also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing systemmay include general purpose central processing units, graphical processing units, application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof.

Storage systemmay comprise any memory device or computer readable storage media readable by processing systemand capable of storing software. Storage systemmay include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read only memory, magnetic disks, optical disks, optical media, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage media. In no case is the computer readable storage media a transitory form of signal transmission (often referred to as “signals per se”), such as a propagating electrical or electromagnetic signal or carrier wave.

In addition to computer readable storage media, in some implementations storage systemmay also include computer readable communication media over which at least some of softwaremay be communicated internally or externally. Storage systemmay be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems co-located or distributed relative to each other. Storage systemmay comprise additional elements, such as a controller, capable of communicating with processing systemor possibly other systems.

Software(including processamong other functions) may be implemented in program instructions may, when executed by processing system, direct processing systemto operate as described with respect to the various operational scenarios, sequences, and processes illustrated herein. For example, softwaremay include program instructions for receiving pod information about pods orchestrated by a container orchestration platform, configuring service objects provided by the container orchestration platform to each correspond to respective ones of the pods based on the information, receive updated pod information indicating a standby pod is now active with an application identifier of a failed pod, and reconfigure a service object associated with the application identifier to correspond to the newly activated pod instead of the failed pod.

In particular, the program instructions may include various components or modules that cooperate or otherwise interact to carry out the various processes and operational scenarios described herein. The various components or modules may be embodied in compiled or interpreted instructions, or in some other variation or combination of instructions. The various components or modules may be executed in a synchronous or asynchronous manner, serially or in parallel, in a single threaded environment or multi-threaded, or in accordance with any other suitable execution paradigm, variation, or combination thereof. Softwaremay include additional processes, programs, or components, such as operating system software, virtualization software, or other application software. Softwaremay also comprise firmware or some other form of machine-readable processing instructions executable by processing system.

In general, softwaremay, when loaded into processing systemand executed, transform a suitable apparatus, system, or device (of which computing systemis representative) overall from a general-purpose computing system into a special-purpose computing system customized to act, for example, as a connection balancer or the like as described herein. Indeed, encoding softwareon storage systemmay transform the physical structure of storage system. The specific transformation of the physical structure may depend on various factors in different implementations of this description. Examples of such factors may include, but are not limited to, the technology used to implement the storage media of storage systemand whether the computer-storage media are characterized as primary or secondary storage, as well as other factors.

For example, if the computer readable storage media are implemented as semiconductor-based memory, softwaremay transform the physical state of the semiconductor memory when the program instructions are encoded therein, such as by transforming the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. A similar transformation may occur with respect to magnetic or optical media. Other transformations of physical media are possible without departing from the scope of the present description, with the foregoing examples provided only to facilitate the present discussion.

Communication interface systemmay include communication connections and devices that allow for communication with other computing systems (not shown) over communication networks (not shown). Examples of communication connections and devices that together allow for inter-system communication may include network interface cards, antennas, power amplifiers, radio-frequency (RF) circuitry, transceivers, and other communication circuitry. The communication connections and devices may communicate over communication media to exchange communications with other computing systems or networks of systems, such as metal, glass, air, or any other suitable communication media.

Communication between computing systemand other computing systems (not shown), may occur over a communication network or networks and in accordance with various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, internets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software defined networks, data center buses and backplanes, or any other type of network, combination of network, or variation thereof.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method, computer program product, and other configurable systems. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more memory devices or computer readable medium(s) having computer readable program code embodied thereon.

Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising.” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” As used herein, the terms “connected.” “coupled,” or any variant thereof means any connection or coupling, either direct or indirect, between two or more elements; the coupling or connection between the elements can be physical, logical, or a combination thereof. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word “or,” in reference to a list of two or more items, covers all the following interpretations of the word: any of the items in the list, all the items in the list, and any combination of the items in the list.

The phrases “in some embodiments,” “according to some embodiments,” “in the embodiments shown,” “in other embodiments,” and the like generally mean the particular feature, structure, or characteristic following the phrase is included in at least one implementation of the present technology and may be included in more than one implementation. In addition, such phrases do not necessarily refer to the same embodiments or different embodiments.

The above Detailed Description of examples of the technology is not intended to be exhaustive or to limit the technology to the precise form disclosed above. While specific examples for the technology are described above for illustrative purposes, various equivalent modifications are possible within the scope of the technology, as those skilled in the relevant art will recognize. For example, while processes or blocks are presented in a given order, alternative implementations may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed or implemented in parallel or may be performed at different times. Further any specific numbers noted herein are only examples: alternative implementations may employ differing values or ranges.

The teachings of the technology provided herein can be applied to other systems, not necessarily the system described above. The elements and acts of the various examples described above can be combined to provide further implementations of the technology. Some alternative implementations of the technology may include not only additional elements to those implementations noted above, but also may include fewer elements.

These and other changes can be made to the technology in light of the above Detailed Description. While the above description describes certain examples of the technology, and describes the best mode contemplated, no matter how detailed the above appears in text, the technology can be practiced in many ways. Details of the system may vary considerably in its specific implementation, while still being encompassed by the technology disclosed herein. As noted above, particular terminology used when describing certain features or aspects of the technology should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific examples disclosed in the specification, unless the above Detailed Description section explicitly defines such terms. Accordingly, the actual scope of the technology encompasses not only the disclosed examples, but also all equivalent ways of practicing or implementing the technology under the claims.

To reduce the number of claims, certain aspects of the technology are presented below in certain claim forms, but the applicant contemplates the various aspects of the technology in any number of claim forms. For example, while only one aspect of the technology is recited as a computer-readable medium claim, other aspects may likewise be embodied as a computer-readable medium claim, or in other forms, such as being embodied in a means-plus-function claim. Any claims intended to be treated under 35 U.S.C. § 112 (f) will begin with the words “means for” but use of the term “for” in any other context is not intended to invoke treatment under 35 U.S.C. § 112 (f). Accordingly, the applicant reserves the right to pursue additional claims after filing this application to pursue such additional claim forms, in either this application or in a continuing application.