Dynamic Management Of Node Headroom To Accommodate Startup Spikes

This application claims priority to U.S. Provisional Patent Application No. 63/737,350 titled “DYNAMIC MANAGEMENT OF NODE HEADROOM TO ACCOMMODATE STARTUP SPIKES,” filed Dec. 20, 2024, the contents of which is incorporated by reference in its entirety for all purposes.

Containerized applications running in a compute cluster (e.g., a Kubernetes cluster) frequently exhibit increased resource demands for CPU and memory during their startup phase, before settling into a lower steady-state usage. For example, applications may experience significant CPU overhead on startup due to operations like class loading, caching large amounts of data, and initializing background services. These temporary resource demands, if not adequately accounted for, can cause the startup of new pods to fail or become severely throttled in highly utilized clusters.

Balancing this initial surge against the generally lower steady-state resource usage presents challenges in workload orchestration. If CPU or memory request values (identifying the minimum reservation for containerized applications or pods) are set high enough to handle startup peaks, those resources may sit underutilized the rest of the time, driving up costs and wasting capacity of nodes in the cluster. Conversely, setting request values to align with steady-state operations may lead to resource contention when multiple applications attempt to start at the same time, causing throttling and even timeouts or crashes.

The disclosure describes a node management service that leverages extended resource configurations to flexibly accommodate both startup and steady-state resource demands for workloads in a compute cluster. In some implementations, the node management service determines a first request value for a pod (corresponding to a steady state usage) and a second request value (corresponding to initialization usage). The node management service provides these request values to the cluster's control plane for scheduling. Once the pod's initialization on the node is complete, the node management service updates the node's capacity metadata to reflect resource availability.

Adjusting the extended resource capacity can involve setting the node's capacity value to a value that may exceed the node's actual capacity. This effectively frees up the “headroom” initially reserved for the pod's startup phase, allowing the scheduler to consider this headroom when placing future pods. Meanwhile, the native CPU request (the first request value) of the pod can be configured at a lower value to only consume the CPU needed for the pod's steady-state usage. In this manner, the node management service prevents unnecessary overprovisioning while providing that initialization phases of newly deployed pods are provided necessary resources.

Various embodiments of the present technology enable dynamic resource allocation for applications running in workload orchestration platforms (e.g., Kubernetes). Historically, each workload's request values (e.g., an amount of CPU or memory reserved for pods in the workload) are specified in a workload configuration, and the orchestration platform's scheduler matches these request values to a node with sufficient capacity. These request values may be set manually by developers in the workload configuration or may be automatically calculated and implemented in the configuration in a process referred to as “right-sizing.”

Common approaches to setting request values for workloads (right-sizing) are as follows: 1) setting request values to the peak demand during startup, or 2) setting request values to the steady-state demand. Some applications (e.g., those written in Java programming language) have much higher resource demand on startup compared to steady state. When the workload's request value (e.g., for CPU and memory) is set to the higher resource demand during startup, the workload may be over-provisioned most of the time (e.g., resources are reserved for a peak demand of an application and will not be fully utilized during other parts of the application's lifecycle). Consider an application that needs 4 CPUs and 8 GB of memory during startup but only 2 CPUs and 4 GB of memory during steady state. If the resources are allocated based on the startup needs, the application will reserve the extra 2 CPUs and 4 GB of memory during steady state without actively using these resources, creating inefficiencies due to overprovisioning.

When request values are set to the steady-state demand, the application is likely to get throttled during startup due to resource congestion. As a result, the application may even fail to start (e.g., due to timeouts). For example, an application that requires 4 CPUs during startup but is only allocated 2 CPUs based on steady-state needs might fail to initialize properly, due to the node not having sufficient available capacity to accommodate the initialization. With fixed resource allocation, existing systems fail to avoid both over-provisioning during steady-state and under-provisioning during startup.

In contrast, various implementations of the technology described herein provide for dynamic resource allocation to accommodate fluctuations in an application's resource demands throughout the application's lifecycle. The disclosure describes a node management service that leverages extended resource definitions (additional resource types recognized by the orchestration platform) to allow workloads to be configured with separate request values for the high resource usage phases of a pod (such as initialization or periods of high demand for applications in the workload) and low resource usage phases (such as steady-state operations or periods of low demand for applications in the workload). Each workload in a compute cluster is defined with both request values for standard resources and request values for extended resources. These request values identify minimum resources for containerized applications (i.e., “pods”) in the workload. The standard resource request values (e.g., CPU and memory) may be set to reflect the workload's usage during low-usage phases such as steady-state operations. The extended resource definitions (e.g., CPU with headroom or Memory with headroom as described herein) may be configured to reflect the increased resource usage during high-usage phases such as initialization (e.g., a workload could have a CPU request of 4 and a CPU with headroom request of 8, as illustrated in Workload 3 in of Table 1 below). The request values are defined in metadata of a configuration file (e.g., a YAML file in Kubernetes), an example of which is illustrated below.

resources:  requests:    cpu: “1”    memory: “100M”    cpuWithHeadroom: “2”    memoryWithHeadroom: “200M”

In the Example Workload Configuration, the workload requests include 1 CPU and 100 MB of memory to accommodate steady-state operations. The workload further requests 2 “CPU with headroom” and 200 MB of “Memory with headroom” to accommodate increased demand during startup. These resources define the minimum available resources for each resource type (CPU, memory, CPU with headroom, and Memory with headroom) on a compute node for deploying the workload (e.g., pods in the workload) to the compute node. Various workloads may have different request values to accommodate the resource needs of containerized applications in the workload. For example, some workloads may need additional CPU and memory during startup and may thus request additional CPU/Memory with headroom, as illustrates with respect to Workloads 1 and 2 in Table 1 below. Other workloads may not utilize additional resources during startup, and may thus have matching request values (e.g., between CPU and CPU with headroom), as illustrated in Workload 3 of Table 1 below.

TABLE 1 CPU CPU w/ Memory Memory w/ Workload request headroom (MB) headroom (MB) 1 4 8 200 300 2 1 2 100 200 3 1 1 200 200

Once the pods in the workload are deployed to nodes in the compute cluster, the node management service monitors lifecycle events of the pods. For instance, a pod might transition from high-demand phases of its lifecycle (such as initialization or a period of high demand for the applications in the workload), to a lower-demand phase (e.g., steady-state operations after initialization or a period of lower demand for the applications in the workload).

Upon detecting that a pod has transitioned from a high-demand phase (e.g., initialization) to a lower demand phase (e.g., steady-state operations), the node management service retrieves the pod's request values for the extended resource and adjusts the node's resource capacity accordingly. In other words, once an application transitions from startup to steady-state, the system reclaims the unused “headroom” and updates the node's metadata to make that headroom available for scheduling other pods; (specifically, by increasing the node's capacity for the extended resource by the difference between the extended resource request value and the steady-state request value). The scheduler of the orchestration platform (e.g., the Kubernetes scheduler), in turn, relies on this updated metadata to make informed decisions when placing new pods in the workload. In this manner, dynamic resource allocation ensures that applications have the additional resources they need only when they need them, enhancing efficiency and avoiding both overprovisioning and under-provisioning.

6 Various embodiments of the present technology provide for a wide range of technical effects, advantages, and/or improvements to computing systems and components. For example, various embodiments may include one or more of the following technical effects, advantages, and/or improvements: 1) non-routine and unconventional dynamic implementation of a node management service; 2) non-routine and unconventional operations for setting request values for workloads in a cluster; 3) use of an extended resource type (e.g., CPU with headroom) which can be dynamically adjusted to allocate resources and meet changing resource needs; 4) dynamic modification of resource capacity metadata for nodes in a cluster; 5) non-routine and unconventional use of extended resources to improve the efficiency of system resource utilization during all stages of an application lifecycle; and/or) non-routine and unconventional use of extended resource definitions in a compute cluster.

1 FIG. 100 100 110 125 120 140 110 125 140 125 120 100 illustrates compute environmentin an implementation. Compute environmentincludes node management service, control plane, compute clusterand compute provider. Node management serviceis in communication with control planeand compute provider. Control planeoperates in compute cluster. However, it is noted that in various implementations compute environmentmay include different or additional components not listed here for brevity.

110 127 128 129 120 110 801 110 140 140 110 110 120 130 125 110 115 117 110 8 FIG. Node management serviceis representative of a software service for managing compute nodes,,in compute cluster. Node management servicemay be a cloud-based service or may operate from one or more servers, which may be represented by computing systemof. Node management servicemay be integrated into compute provider(e.g., as a service offered by compute provider) in some implementations. In other implementations, node management servicemay be a third-party or standalone service. Node management serviceinterfaces with compute clustervia API serverof control plane. Node management serviceincludes control plane interfaceand resource manager. However, it is noted that in various implementations, node management servicemay include different or additional components not listed here for brevity.

115 130 125 115 130 120 125 115 130 110 133 Control plane interfaceis representative of a service such as an API layer that interfaces with API serverof control plane. Control plane interfaceis configured to obtain scale-up requests from API server, including requests to scale up new workloads (i.e., a group of one or more pods hosting containerized applications) in compute cluster. In some implementations, the scale-up request identifies the name of the workload, the desired number of pod replicas. This scale-up request may be triggered, for example, by a pod autoscaler of control plane(e.g., a Horizontal Pod Autoscaler of Kubernetes). Control plane interfacereceives such requests from API server, and subsequently relays them to node management serviceso the appropriate resource allocations can be determined and updated in key-value store.

115 130 133 Control plane interfaceis also configured to provide workload request values to API serverfor updating key-value store. These request values may include native resource request values for a workload such as CPU and memory. The request values may also include resource request values for one or more defined extended resources that indicate resource requirements for pods during initialization or startup. The extended resources may define the CPU or memory (or both) utilization during initialization of a pod, which may be higher than its steady-state utilization of CPU. Accordingly, the request values for a given pod may include both natively defined request values (e.g., for CPU and memory) request value and extended-resource request values (e.g., CPU with headroom and Memory with headroom as described further below). The native request values (CPU and memory) identify resource utilization of pods during steady-state operations and the extended resource request values (CPU with headroom and Memory with headroom) identify resource utilization of the pods during initialization.

125 131 131 127 128 129 Extended resources, as defined for example by Kubernetes, allow cluster operators and third-party providers to introduce resource types beyond the natively defined CPU and memory resources. In Kubernetes, CPU and memory are intrinsic (native) resource types known and managed by control plane, representing fundamental compute and storage capabilities of a node. CPU units correspond to available cores or vCPUs, and memory corresponds to available RAM. Together, these native resources form the baseline against which schedulermeasures workload demands. Extended resources build upon this baseline by defining additional consumable quantities that schedulercan consider when placing pods on compute nodes,,. The extended resources may define CPU utilization during a pod's startup or initialization period (referred to herein as “CPU with headroom”), the memory utilization during a pod's startup (referred to herein as “Memory with headroom”), or both. Additionally, while extended resources for CPU and memory are described herein, it is noted that the extended resource may alternatively or additionally include other resource types such as GPU in various implementations.

115 130 133 127 127 133 127 127 165 127 128 129 110 117 400 4 4 FIGS.A andB Control plane interfacealso provides, to API server, requests for updates to metadata in key-value storeassociated with compute nodes. The metadata of a compute nodein key-value storecontains several values regarding the capacity and state of the node. For example, it may include the total allocatable CPU capacity for both native and extended resources, the free capacity for both native and extended resources, and labels or annotations indicating the operational state of nodes (e.g., “initialization phase,” “ready for deployment” when compute nodeis ready to accept new pods, “disk pressure” when compute nodehas low disk capacity, and “memory pressure” where compute node has low memory). Node agentidentifies these various states on compute nodes,,and updates key-value store with the detected states. Node management servicealso updates to the metadata to define the nodes' capacity of the extended resource (e.g., CPU with headroom and/or Memory with headroom). These changes are determined by resource manager, as explained further below. The dynamic adjustment of the node metadata is also illustrated and described further below with respect to workflowin.

117 110 120 117 117 117 117 Resource manageris representative of a controller in node management servicethat manages resources in compute cluster. Resource manageris configured to perform right-sizing functions for pod resource requests, including setting request values for both native resources (CPU and memory) and extended resources (CPU with headroom and/or Memory with headroom) for workloads. Resource managerperforms right-sizing by calculating request values for a workload that strikes a balance between over-provisioning and under-provisioning. This calculation may be based on historical resource usage data of the application or applications in the pods. Resource managermay right-size both the native CPU resource and the extended CPU resource (CPU with headroom and/or Memory with headroom) by analyzing historical resource usage throughout the lifecycle of pods in the workload in previous deployments. In particular, resource managermay calculate the native CPU request value based on historical steady-state usage of applications, and the extended CPU request value (CPU with headroom and/or Memory with headroom) based on the observed or anticipated startup resource demands. However, in some cases an application owner or user may also manually define these request values when setting a workload configuration.

117 127 120 165 127 133 110 133 117 Resource manageris also configured to determine that initialization of pods on compute nodesis complete. This may include, for example, receiving an indication from an application running in the pod that the application is ready to handle external requests or web traffic, by monitoring a readiness probe in compute cluster. Such readiness probes may be executed by node agent(e.g., Kubelet) on compute nodes, which updates a readiness status of a pod in key-value storewhen the pod is ready to handle external traffic. Node management servicemay in turn monitor the readiness status of pods in key-value storeto determine when pods have completed the initialization phase. In other implementations the determination that initialization is complete may be based on elapse of a predetermined time period (e.g., 1 minute, 5 minutes, 10 minutes, etc.) since deployment of the pod to a node. This time period may be determined based on the time applications on a given pod take to initialize and may vary depending on the initialization time of a particular pod. In other implementations, resource managermay utilize various other methods to determine that the initialization is complete.

117 127 133 115 117 127 128 129 117 131 400 117 127 4 4 FIGS.A andB 4 4 FIGS.A andB Resource manageris further configured to update capacity metadata of the extended resource in compute nodes(in key-value storeand via control plane interface) upon determining that initialization is complete. In particular, resource managermay increase the extended resource metadata (e.g., “CPU with headroom” capacity of a compute node,,) by a difference between the extended resource value for the pod (e.g., the “CPU with headroom”) and the native request value for the pod (e.g., the CPU request aligning with steady-state needs). Thus, resource managersynthetically increases the value of the extended resource such that the schedulermay account for resource availability for handling initialization of future pods. This increase is illustrated and described with respect to workflowof. Resource managermay also decrease the capacity metadata when pods are scaled down from compute nodes, as explained further in the discussion of.

127 127 128 129 117 127 128 129 117 127 131 400 4 4 FIGS.A andB It is noted that the extended resource capacity metadata for a compute node(e.g., CPU with headroom and/or Memory with headroom) does not always align with the actual CPU or memory capacity of the virtual machine or server hosting the node. When compute nodes,,are first initialized, resource managersets the “CPU with headroom” capacity to match the actual CPU capacity of the node (thus matching the node's CPU capacity with its CPU with headroom capacity); and likewise, for “Memory with headroom” according to some implementations. However, as pods with disparate request values are added to a compute node,, or, resource managerincreases the CPU with headroom capacity metadata (or Memory with headroom capacity data) above the actual capacity of compute nodesto signal resource availability. This is performed because, during steady state after a pod is initialized, it reserves more “CPU with headroom” and/or “Memory with headroom” than the CPU/memory actually utilized by the pod. Increasing the “CPU with headroom” and/or “Memory with headroom” capacity accounts for this discrepancy, signaling to schedulerthat “headroom” is available to accommodate the initialization of future pods. Accordingly, “CPU with headroom” and “Memory with headroom” are understood as an extended or synthetic resources of a node that do not necessarily reflect the actual operational compute reservations of the node. This concept is illustrated and explained further below in the discussion of workflowof.

117 127 128 129 125 125 In various embodiments, resource managerdynamically updates node metadata representing node resource capacity during different phases of the application life cycle for pods deployed to compute nodes,,. This update to the node capacity effectively “tricks” control planeto allow for more efficient resource utilization. For example, if an initial phase requires more resources than a later phase, control planecan assign workloads based on the higher initial resource needs and then increase the node's available capacity as resource usage decreases in the later phase.

125 120 125 710 125 120 125 125 131 133 135 125 7 FIG. Control planerepresents an orchestration platform that manages and coordinates the resources within compute cluster. Control planemay be represented by control planeof. Control planemay operate in one or more nodes of compute cluster. Control planemay be a Kubernetes control plane in some implementations. However, it is noted that the concepts described herein are not limited to Kubernetes and may be applied to other orchestration platforms. Control planeincludes scheduler, key-value store, and controller manager. However, it is noted that in various implementations, control planemay include different or additional components not listed here for brevity.

131 127 131 133 131 127 131 127 128 129 131 133 131 127 128 129 127 128 129 131 127 128 129 131 740 7 FIG. Scheduleris configured to schedule pending pods to compute nodes. Schedulerreads data from key-value storeto identify created pods that are ready for scheduling. Scheduleridentifies compute nodesthat have sufficient available compute capacity to accommodate the request values of the pending pods, including native resource request values (CPU and memory) and extended resource request values such as CPU with headroom and Memory with headroom. To make this determination, schedulerdetermines the “free capacity” for each resource type on compute nodes,,. Specifically, schedulertakes the total capacity for each resource type and subtracts the amount being consumed by pods deployed to the compute nodes (determined based on the request values). The relevant data for performing this calculation may be retrieved from key-value store. Schedulerbinds each pending pod with a compute node,, orhaving sufficient capacity for each requested resource. The increase of the “CPU with headroom” capacity metadata for compute nodes,,provides schedulerwith an indication of which compute nodes,,have sufficient available capacity to accommodate the initialization phase of pending pods. Schedulermay be represented by schedulerof.

133 120 133 127 133 127 110 133 735 7 FIG. Key-value storerepresents a data store that holds parameters used to orchestrate and manage the state of compute clusterincluding the lifecycle of pods in the cluster. Key-value storecontains request values for the pods (for both native CPU and memory resources and extended resources such as CPU with headroom and Memory with headroom) as well as the readiness status of pods as indicated by pod readiness probes in compute nodes. Key-value storealso includes resource capacity metadata for compute nodes, including each node's capacity for native resources (CPU and memory) and extended resources such as CPU with headroom. Node management serviceupdates the metadata for the CPU with headroom capacity for nodes after pod initialization, as described herein. Key-value storemay be represented by key-value storeof.

135 120 133 135 133 110 135 131 135 135 730 7 FIG. Controller managerrepresents a suite of controllers that continuously observe the state of compute clusterthrough key-value storeand adjust cluster operations to match desired configurations. For example, if a scale-up request is issued (e.g., in response to increased workload), controller managerinitiates the creation of new pods by updating the cluster state through key-value store. As the node management serviceupdates the “CPU with headroom” capacity after pod initialization, controller managercan trigger scheduling events that capitalize on the newly freed initialization-phase capacity. By coordinating with schedulerand leveraging updated node capacity metadata (including the extended resources described herein), controller manageraccommodates both steady-state and startup CPU utilization. Controller managermay be represented by controller managerof.

120 127 128 129 125 120 125 127 128 129 120 127 128 129 120 120 700 1 FIG. 7 FIG. Compute clusteris representative of a cluster of compute nodes,,orchestrated by control plane. Compute clustermay be, for example, a Kubernetes cluster, however it is noted that the described technology is applicable to clusters in other orchestration platforms. Control plane, described above, may be a service operating in one or more compute nodes,,of compute cluster. While three compute nodes,,are shown infor simplicity, compute clustermay include any number of compute nodes. Compute clustermay be represented by compute clusterof.

127 128 129 801 140 127 128 129 8 FIG. Compute nodes,,may be implemented in virtual machines or physical servers (e.g., computing systemshown in), depending on the deployment. These resources may be provisioned from compute provider, which operates the physical infrastructure (including data centers) that provides the computing resources for hosting compute nodes,,.

1 FIG. 1 FIG. 127 127 128 129 120 127 161 163 165 127 167 127 131 127 161 163 127 167 161 163 127 127 161 163 also illustrates an expanded view of compute node, which may be representative of each compute node,,of compute cluster. Compute nodemay be a virtual machine for running processes, including pods,, and node agent. While two pods are illustrated infor brevity, compute nodemay run any number of pods. Free capacityrepresents currently unutilized resources of compute node, allowing schedulerto schedule additional pods to compute node. Each time a pod (e.g., pods,) is deployed to compute node, the free capacityfor each resource type decreases by the amount specified in the request value of the pod,. However, increasing the overall capacity of the extended resource (as described herein) on compute nodesynthetically makes available more free capacity (“headroom”) for scheduling new pods to compute node. Increasing the overall capacity of compute node for an extended resource (CPU with headroom or Memory with headroom) provides that organizations can reserve more resources during initialization without “using up” the full amount of this resource for the entire lifecycle of the pod,.

161 163 120 117 120 117 127 128 129 125 127 128 129 120 Pods,, as referenced herein, are deployable units of computing within a container orchestration platform such as Kubernetes. Each pod encapsulates one or more containerized applications. A workload may consist of multiple pods (e.g., pod replicas in a ReplicaSet of Kubernetes) that run concurrently in compute clusterto meet usage demands. Resource managercollects information about pods running in compute clusterto estimate a level of disruptions of the workload, including violations of graceful termination and minimum capacity requirements. Resource managerutilizes this data to make informed decisions about whether resource allocations need to be updated, as described in greater detail herein. Pods are scheduled to compute nodes,,by control plane. Each compute node,,in compute clustermay host one or more pods from various workloads.

165 127 161 163 165 161 163 133 117 127 128 129 Node agent, (e.g., Kubernetes Kubelet), is configured to report on the status of processes running on compute node(e.g., pods,). Node agentidentifies the status of pods,(e.g., whether it is starting up, running, deleted, etc.) and updates the status in key-value store. Resource managerin turn utilizes this information to determine when startup is complete to adjust the extended resource capacity of compute nodes,,(e.g., CPU with headroom and Memory with headroom).

127 128 129 140 120 117 127 128 129 140 In various implementations, compute nodes,,may vary in resource configurations and availability. For example, some compute nodes offered by compute providermay have a larger memory footprint (e.g., “high-memory” configurations), whereas others may offer more CPU (“high-CPU” configurations). Because compute clustermay contain an assortment of these node types, resource manageris responsible for dynamically matching workloads to the most suitable node based on current capacity, resource requirements, and node characteristics. Examples of resource availability for various compute nodes are illustrated in Table 2 below. It is noted that Table 2 provides a selection of types of compute nodes,,for illustrative purposes; compute providermay offer a wide range of instance types for compute nodes with different resource configurations.

TABLE 2 Compute Node Type CPUs Memory (GB) General Purpose Node 4 8 High CPU Node 16 8 High Memory Node 4 16

140 127 128 129 120 110 140 140 110 140 Compute providerrepresents a source of compute resources, including preemptible and non-preemptible virtual machines (VMs) that host compute nodes,,. These VMs may be operated from servers distributed across one or more geographic locations and are made available to customers, such as workload application owners, for use in compute clusters like compute cluster. In some implementations, node management servicemay be provided by compute provider, in which case it is integrated into compute provider. In other implementations, node management servicemay be offered by a third party, independent of compute provider. Examples of compute providers include Amazon Web Services, Google Cloud, IBM Cloud, and others.

2 FIG. 8 FIG. 2 FIG. 110 200 200 801 200 illustrates a resource management process performed by node management service, represented by process. Processis employed by a computing device to provide resource management, an example of which is provided by computing systemof. Processmay be implemented in program instructions (software and/or firmware) by one or more processors of the computing device. The program instructions direct the computing device to operate as follows, referring parenthetically to the steps in.

110 120 201 161 163 120 125 125 120 125 120 Node management serviceidentifies a workload to scale up in compute cluster(step). The workload may consist of one or more pods, such as pods,, to be deployed to compute cluster. Identifying the workload may include receiving an instruction to scale-up the workload from control plane(e.g., to meet increased usage demands), in some implementations. This instruction can be triggered by control planebased on metrics such as CPU utilization, memory usage, or incoming request throughput. For example, if existing pods in compute clusterare experiencing high usage levels (e.g., sustained CPU load above a defined threshold), control planemay determine to scale up additional replicas to meet usage demands. In other implementations, the identification of the workload may occur when a user submits a request to deploy a new application or service into compute cluster.

110 203 125 Node management servicedefines the first request value and the second request value for the workload (step). The first request value is associated with a resource usage of a pod in the workload during a low-resource usage phase (e.g., steady-state operations or periods of low demand for an application hosted on the pod), and the second request value is associated with the resource usage of the pod during a high-usage phase (e.g., initialization of the pod or periods of high demand for the application). The first request value may be associated with the native resource definition utilized by control plane(e.g., for CPU or memory), and the second request value may be associated with an extended resource definition (e.g., “CPU with headroom” or “Memory with headroom” as described herein). Accordingly, the native CPU resource is used to define the pod's resource utilization during low-demand periods such as steady state, while the extended resource is used for the pod's utilization of CPU resources during high demand periods such as initialization or startup. The request values may be defined in workload configuration metadata or pod configuration metadata.

110 To define the request values, node management servicemay perform automatic “right-sizing” in some implementations. This process may include analyzing historical resource usage of the pod in previous deployments. For instance, the service can employ machine learning algorithms to predict optimal resource requirements by examining past CPU and memory consumption patterns during both high demand (e.g., startup) and low demand (e.g., steady-state) phases. In other scenarios, defining the request values may include identifying manually configured request values (e.g., a developer may manually set the request values for CPU, memory, CPU with Headroom, and/or Memory with headroom in the workload configuration metadata).

200 It is noted that the concepts described with respect to processare not limited to startup and initialization; in other implementations, the first and second request values may be associated with various other phases in an application's lifecycle. For example, the first request value may be associated with “Phase One” operations (e.g., periods of increased application usage or initialization periods) and “Phase Two” operations, which may include periods of lighter demand (e.g., periods of decreased application usage or steady-state operations).

110 125 205 110 133 131 127 127 128 129 133 Node management serviceprovides the first request value and the second request value to control plane(step). Specifically, node management serviceupdates key-value storewith the first request value and the second request value. Schedulerthen utilizes these request values to schedule the pod to a compute nodebased at least in part on the first request value and the second request value of the pod, as well as the node capacity metadata for compute node,, orretrieved from key-value store.

125 127 120 207 131 127 133 167 127 125 165 127 1 FIG. Control planedeploys the pod to a compute nodein compute clusterhaving sufficient capacity to accommodate the request values (step). Schedulerchecks metadata for pending pods and compute nodesin key-value storeto identify those compute nodes with sufficient free capacity (e.g., free capacityof) for each resource type (e.g., CPU, Memory, CPU with headroom, and Memory with headroom) of the pending pod. Once a compute nodeis selected, control planeinstructs node agentto instantiate and run pods' containerized application(s) on compute node.

110 209 110 Node management servicechecks the status for the pod (step). In some implementations, node management servicecan leverage pod lifecycle events (e.g., indications from readiness probes) to check the current state of the application. Readiness probes are mechanisms configured on a pod that check whether an application is ready to handle external requests. When a readiness probe succeeds, it indicates that the application within the pod has completed its startup tasks and is now ready to serve traffic, effectively signaling the end of the initialization phase. In other implementations, checking the status may include determining whether a predetermined time-period (e.g., 1 minute, 5 minutes, 10 minutes, etc.) has elapsed since the start of deployment of the pod to the node.

110 127 211 209 200 213 200 209 Node management servicedetermines if initialization of the pod on compute nodeis complete (step). If the pod status (as identified in step) indicates that initialization is complete (e.g., the readiness probe indicates that the application is ready to handle requests, or the predetermined time-period is complete), processproceeds to step. If initialization is not complete (e.g., the readiness probe indicates that the application is still initializing, or the predetermined time-period has elapsed), processreturns to stepfor continued monitoring.

211 110 120 110 In other implementations, stepmay include detecting a transition of the pod from any high-usage phase (such as periods of high demand of the application) to the low-usage phase (such as periods of low demand for the application). To accomplish this, node management servicemay continuously collect and analyze metrics (e.g., CPU utilization, memory consumption, or network throughput) from pods in compute cluster. When these metrics fall below a predefined threshold (such as a certain percentage of CPU usage over a rolling time window) node management serviceidentifies that the pod has shifted into the low-usage phase.

110 127 128 129 125 213 110 133 110 127 Node management serviceupdates the capacity metadata for compute node,,in control plane(step). Specifically, node management serviceincreases the extended resource (e.g., CPU with headroom and/or Memory with headroom) capacity metadata in key-value store. To calculate the new capacity for the extended resource, node management serviceretrieves the first and second request values for the deployed pod (e.g., from the pod's workload configuration metadata). It then takes a difference between the pod's second request value (for the extended resource such as CPU with headroom or Memory with headroom) and the pod's first request value (for the native CPU or memory resource). It then adds this difference to the current capacity of compute nodefor the extended resource (CPU with headroom or Memory with headroom).

127 211 110 127 127 For example, consider a pod that has a CPU request value of 3 (reflecting steady-state usage) and a “CPU with headroom” request value of 4 (reflecting increased initialization usage). The pod has been deployed to a compute nodethat has a total capacity of 8 for the native CPU resource and a total capacity of 8 for the extended resource “CPU with headroom.” After determining that the initialization period for the pod is complete (at step), node management servicecalculates the updated capacity for compute nodeby taking between the pod's “CPU with headroom” request value and CPU request value as follows: 4−3=1. Node management service increases the node's capacity for “CPU with headroom” by 1 (the difference) to arrive at 9. Thus, after initialization, the total capacity of compute nodefor “CPU with headroom” is updated to 9 (8+1); while the total capacity for “CPU” (the natively defined resource) remains static at 8.

125 215 131 133 127 128 129 131 Control planeutilizes the updated node capacity metadata to make scheduling decisions for additional pending pods (step). Schedulerretrieves, from key-value store, the request values for the additional pending pods and the free capacity of compute nodes,,for both native and extended resources (e.g., CPU with headroom and Memory with headroom). Schedulerschedules pending pods that have sufficient free capacity for each resource type (e.g., CPU, Memory, CPU with Headroom, and/or Memory with headroom). Updating the capacity value for the extended resource definition (e.g., CPU with headroom and/or Memory with headroom) thus frees up room for the scheduling of these additional pods.

127 128 129 This adjustment process enhances resource utilization and scheduling efficiency by reclaiming unused headroom once a pod completes its initialization phase. By recalculating and increasing the extended resource capacity for compute nodes,,based on the difference between initialization and steady-state demands, the system ensures that available resources reflect real-time availability without overprovisioning.

3 FIG. 1 FIG. 200 100 300 300 110 133 131 135 300 130 illustrates an operation sequence of an application of processin the context of compute environmentin an implementation, represented by sequence. Sequenceincludes node management service, key-value store, scheduler, and controller manager. Various operations in sequencemay be performed via API serverof.

300 110 133 120 110 135 120 110 110 In sequence, node management servicesets, in key-value store, initial node capacity metadata for nodes in compute cluster(which may include initializing the “CPU with headroom” capacity to match the native CPU resource capacity and/or initializing the “Memory with headroom” capacity to match the native memory resource capacity). Node management serviceobtains a scale-up request from controller manager. This scale-up request may identify new pods to scale-up in compute cluster(for example based on increased demand for an application or a new deployment). Node management serviceright-sizes the workload, including setting both native resource request values (CPU and memory) and extended resource request values (e.g., CPU with headroom and Memory with headroom). However, it is noted that in some implementations, the request values may be user-defined or a combination of user-defined and system defined (e.g., a user may define the “CPU with headroom” request while node management servicedefines the CPU request, or vice versa).

110 133 131 133 127 128 129 110 127 110 133 110 131 133 215 200 Node management serviceupdates key-value storewith the request values for the pods. Schedulerobtains scheduling metadata from key-value store(including the request values and the CPU capacity values) and schedules pods to compute nodes,,based at least in part on the nodes having sufficient free capacity for each resource (including native and extended) to accommodate the request values of the pod. Node management servicedetermines that initialization of the pod on compute nodesis complete after deployment (thus identifying that the pod is no longer actively utilizing the full “CPU with headroom resource” request amount). Node management serviceupdates the node capacity metadata in key-value store. Specifically, node management serviceincreases the “CPU with headroom” resource capacity by a difference between the native CPU request of the pod with the CPU with headroom request. Schedulerretrieves this updated capacity metadata from key-value storeand utilizes this metadata to make scheduling determinations, as discussed above with respect to stepof process.

4 4 FIGS.A andB 4 FIG. 4 4 FIGS.A andB 4 4 FIGS.A andB 4 4 FIGS.A andB 400 400 110 125 400 499 125 120 illustrate workflow, which illustrates a sequence of deployments and removals of pods leveraging the extended resource defined herein. The operations illustrated in workflowmay be performed by node management serviceoperating in conjunction with control plane. Steps in workfloware represented by circled numbers in. The top bar of elements inis representative of a native CPU resource, while the bottom bar is illustrative of the “CPU with headroom” resource, as illustrated in the legend of element.illustrate the native CPU resource as “cpu” and the extended CPU resource as “cpuWithHeadroom,” which may be the naming conventions used in control planeand compute cluster; however, it is noted that other naming conventions may be utilized in various implementations. Further, whileare illustrated with respect to CPU resources, the same concepts described are applicable to other resource types (e.g., the native memory resource and the extended “Memory with headroom” resource described herein).

405 450 127 120 110 450 110 450 1 FIG. Step 1 illustrates deployment of Pod A (represented by element) to a node (represented by element) which may be one of compute nodesin compute clusterof. Pod A has a CPU request of 1 and a “CPU with headroom” request of 1. (Accordingly, Pod A does not request additional resources for its initialization phase). These request values may be determined by node management serviceright-sizing operations, or may be user-defined, as explained above. Elementillustrates the resource state of the node upon deployment of Pod A. The node has a total CPU capacity of 5 (which may represent 5 CPU cores or vCPUs). Further, the node begins with a total “CPU with headroom” capacity of 5 (noting that node management servicemay initialize the node with a “CPU with headroom” capacity equal to the native CPU capacity). Elementillustrates that deployment of Pod A to the node does not trigger a change of the CPU with headroom capacity of the node, since the pod has the same request value (1) for both the “CPU with headroom” resource and the CPU resource. Upon deployment of Pod A, the free capacity for both CPU and CPU with headroom in the node is 4.

410 455 131 455 120 Step 2 illustrates deployment of pod B (represented by element) to the node (represented by element). Pod B has a CPU request of 2 and a “CPU with headroom” request of 4. In scheduling Pod B, scheduleridentifies a node that has free CPU capacity of 2 and free “CPU with headroom” capacity of 4. Elementillustrates that deployment of Pod B to the node consumed the remaining CPU with headroom capacity (making further pods unschedulable to the node during initialization of Pod B and mitigating the chances of congestion), while leaving a free CPU capacity of 2. It is noted that in the described technology, each pod deployed to compute clustermay be defined with a “CPU with headroom” request that is not smaller than its CPU request.

460 415 110 131 205 207 200 460 Step 3 illustrates the modification of the node's CPU with headroom capacity (represented by element) after Pod B has completed initialization (as illustrated in element). Specifically, Pod B may only consume 2 cpu during steady-state, while still reserving 4 “CPU with headroom” with its request value. Accordingly, there are 2 “free” CPU with headroom resources available in this scenario. After initialization of Pod B, node management serviceupdates the CPU with headroom capacity of the node by 2 (the difference between pod B's CPU request and CPU with headroom request) to convey to schedulerthat the node has available “headroom” for accommodating the initialization of future pods (as described above in relation to stepsandof process). In other words, the total CPU with headroom capacity is increased such that the free CPU with headroom capacity (i.e., the available CPU with headroom) matches the free CPU capacity after initialization of a pod is complete (noting that during initialization, the free capacity of these resources does not match, thus avoiding overscheduling on the node during initialization of pods). Elementillustrates that the node has 2 CPU that are not being actively utilized by currently deployed pods.

4 FIG.B 400 420 465 465 460 Continuing in, step 4 of workflowillustrates deployment of Pod C (represented by element) to the node (represented by element). In this case, Pod C has a CPU request of 1 and a CPU with headroom request of 2. The node can accommodate Pod C, as illustrated in element(since in its previous state at elementthe free capacity for both resources was 2). It is noted that absent the increase in CPU with headroom capacity at step 3, the node would not have sufficient CPU with headroom to accommodate pod C, making Pod C unschedulable to the node. Thus, adjusting the CPU with headroom as described herein allows space to be freed up after initialization is complete.

470 425 Step 5 illustrates the modification of the node's CPU with headroom capacity (represented by element) after pod C has completed initialization (as illustrated in element). Similar to the discussion of step 3 above, CPU with headroom capacity is increased by the difference between Pod C's CPU request and its CPU with headroom request (in this case an increase of 1 to arrive at 8 total). Thus, after initialization of Pod C, the node has 1 free CPU and 1 free CPU with headroom.

465 110 475 Step 6 illustrates deletion of Pod C from the node (as illustrated in element). Pods may be removed due to scaling down or reduced application demand. Upon removal of Pod C, node management servicedecreases the node's total CPU with headroom capacity by the difference between the deleted pod's CPU request and “CPU with headroom” request (in this case, 1). This reduction is performed because after deletion, Pod C no longer consumes disparate amounts of CPU and CPU with headroom on the node. Thus, the total CPU with headroom capacity is reduced from 8 to 7 after deletion. Elementalso illustrates that resources are freed up on the node after deletion, resulting in 2 free CPU and 2 free CPU with headroom.

5 FIG. 8 FIG. 5 FIG. 110 500 500 801 500 illustrates a resource management process performed by node management service, represented by process. Processis employed by a computing device to provide resource management, an example of which is provided by computing systemof. Processmay be implemented in program instructions (software and/or firmware) by one or more processors of the computing device. The program instructions direct the computing device to operate as follows, referring parenthetically to the steps in.

110 127 501 110 135 200 500 501 127 200 200 500 Node management servicedetermines that a pod has been removed from compute node(step). Node management servicemay obtain this notification, for example, from controller manager. It is noted that processesandmay be integrated into one overall process in some implementations. For example, the deleted pod of stepmay be the same pod that was previously deployed to compute nodeas described above in process. Processes,each may also operate as independent processes in various implementations.

110 503 133 110 110 Node management serviceretrieves the first request value and the second request value for the pod (step). These values may be retrieved from key-value storein some implementations. In other implementations, node management servicemay maintain the request values in metadata internal to node management service.

110 127 505 110 127 110 127 4 FIG.B Node management servicecalculates, in response to determining that the pod has been removed, an updated capacity value for the compute node(step). Specifically, node management servicedetermines an updated value for the extended resource (e.g., CPU with headroom and/or Memory with headroom) capacity in compute node. To calculate the updated value, node management servicetakes a difference for the deleted pod's first request value (e.g., the pod's request value for a native CPU or Memory resource) and second request value (e.g., the request value for the CPU with headroom resource or the Memory with Headroom resource) and subtracts this difference from the current capacity for the extended resource in compute node. This calculation is illustrated in step 6 of, where “Pod C” is deleted.

110 127 125 507 110 133 503 Node management serviceupdates the capacity metadata for compute nodein control planewith the calculated value (step). Specifically, node management serviceupdates, in key-value store, the node's capacity for the extended resource (CPU with headroom and/or Memory with headroom) with the value calculated in step.

125 509 131 133 127 128 129 131 Control planeutilizes the updated node capacity metadata to make scheduling decisions for additional pending pods (step). Schedulerretrieves, from key-value store, the request values for the additional pending pods and the free capacity of compute nodes,,for both native and extended resources (e.g., CPU with headroom and Memory with headroom). Schedulerschedules pending pods that have sufficient free capacity for each resource type (e.g., CPU, Memory, CPU with Headroom, and/or Memory with headroom). Updating the capacity value for the extended resource definition (e.g., CPU with headroom and/or Memory with headroom) thus frees up room for the scheduling of these additional pods.

6 FIG. 1 FIG. 500 100 600 600 110 133 135 300 130 illustrates an operation sequence of an application of processin the context of compute environmentin an implementation, represented by sequence. Sequenceincludes node management service, key-value store, and controller manager. Various operations in sequencemay be performed via API serverof.

135 110 127 501 500 110 127 503 500 110 133 127 To begin, controller managerprovides, to node management service, a notification that a pod has been deleted from compute node, as discussed above with respect to stepof process. Pod deletion may occur, for example, due to scale-down in response to reduced demand, resource optimization, or failure of the pod or its underlying infrastructure. Node management servicecalculates an updated capacity value for the extended resource (e.g., CPU with headroom) of the compute nodefrom which the pod was deleted, as discussed above in relation to stepof process. Node management serviceupdates, in key-value store, the capacity metadata for the compute nodewith the calculated capacity value for the extended resource. This update may decrease the capacity by a difference between a native request value for the deleted pod (e.g., CPU) and the request value for the extended resource (e.g., CPU with headroom or Memory with headroom). This decrease is representative of the fact that the deleted pod is no longer consuming extra “headroom” upon deletion, and the extended resource is updated accordingly.

7 FIG. 700 700 710 750 700 illustrates compute clusterin an implementation. Compute clusterincludes control planeand compute nodes. Compute clustermay be a Kubernetes cluster; however, it is also representative of various other orchestration platforms.

710 700 710 700 710 720 730 735 740 Control planeis representative of a software service that manages resources in compute cluster, and may, for example, be a Kubernetes control plane. Control planecan operate from one or more nodes or virtual machines within compute cluster. Control planeincludes API server, controller manager, key-value store, and scheduler.

720 710 720 750 730 735 740 720 700 735 720 API serveris a central interface in control planefor processing and validating requests. API serveris in communication with compute nodes, controller manager, key-value store, scheduler, as well as external clients such as a node management service as described herein. API serverprocesses requests (such as requests to create, update, or delete resources), validates them, and updates the state of compute clusterin key-value store. API servermay also handle authentication and authorization of client requests, ensuring that only permitted users and services can access or modify cluster resources.

730 700 735 730 750 Controller manageris representative of a service that manages controllers to maintain the state of compute clusterby continuously monitoring the current state and reconciling it with the desired state as defined in key-value store. Controller managerorchestrates tasks to achieve the desired state, such as coordinating the creation or deletion of pods to match the specified number of pod replicas, monitoring the health of compute nodes, and initiating replacement or recovery actions for failed nodes.

735 Key-value store, which may be Kubernetes etcd in some implementations, maintains the cluster's configuration data and state information.

740 750 Schedulerassigns workloads such as pods to appropriate compute nodes. It makes scheduling decisions based on resource availability, constraints, and policies.

750 750 700 750 755 757 7 FIG. Compute nodesare representative of virtual machines or physical servers on which workloads run. While three compute nodesare shown infor clarity, it is noted that compute clustermay include any number of compute nodes. Compute nodeincludes network proxyand node agent.

755 750 750 Network proxy(e.g., Kube-proxy) is representative of a service running on compute nodethat maintains network rules on compute nodeto facilitate communication between services within the cluster. It manages network routing for service discovery and load balancing, providing that requests to a particular service are directed to one of the corresponding backend pods.

757 750 750 757 720 757 757 750 710 Node agent(e.g., Kubelet) is representative of a service running on compute nodethat manages the state of pods on compute node. Node agentcommunicates with API serverto receive instructions about which pods to run. Node agentperforms tasks such as starting, stopping, and managing containerized workloads. Additionally, node agentmonitors running pods containers, collects resource usage metrics, and reports on the state of compute nodeto control plane.

8 FIG. 801 801 801 illustrates computing system, which is representative of any system or collection of systems in which the various applications, processes, services, and scenarios disclosed herein may be implemented. Examples of computing systeminclude, but are not limited to server computers, web servers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machine, container, and any variation or combination thereof. (In some examples, computing systemmay also be representative of desktop and laptop computers, tablet computers, and the like.)

801 801 802 803 805 807 809 802 803 807 809 Computing systemmay be implemented as a single apparatus, system, or device or may be implemented in a distributed manner as multiple apparatuses, systems, or devices. Computing systemincludes, but is not limited to, processing system, storage system, software, communication interface system, and user interface system. Processing systemis operatively coupled with storage system, communication interface system, and user interface system.

802 805 803 805 806 200 802 805 802 801 Processing systemloads and executes softwarefrom storage system. Softwareincludes and implements resource management processes, which is representative of the processes discussed with respect to the preceding Figures, such as process. When executed by processing system, softwaredirects processing systemto operate as described herein for at least the various processes, operational scenarios, and sequences discussed in the foregoing implementations. Computing systemmay optionally include additional devices, features, or functionality not discussed for purposes of brevity.

8 FIG. 802 805 803 802 Referring still to, processing systemmay include a microprocessor and other circuitry that retrieves and executes softwarefrom storage system. Processing system multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing systeminclude general purpose central processing units, microcontroller units, graphical processing units, application specific processors, integrated circuits, application specific integrated circuits, and logic devices, as well as any other type of processing device, combinations, or variations thereof.

803 802 805 803 803 803 802 Storage systemmay comprise any computer readable storage media readable by processing systemand capable of storing software. Storage systemmay include volatile and nonvolatile, removable, and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read only memory, magnetic disks, optical disks, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage media. In no case is the computer readable storage media a propagated signal. Storage systemmay be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems co-located or distributed relative to each other. Storage systemmay comprise additional elements, such as a controller capable of communicating with processing systemor possibly other systems.

805 806 802 802 805 Software(including resource management processes) may be implemented in program instructions and among other functions may, when executed by processing system, direct processing systemto operate as described with respect to the various operational scenarios, sequences, and processes illustrated herein. For example, softwaremay include program instructions for implementing resource management processes and procedures as described herein.

Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” As used herein, the terms “connected,” “coupled,” or any variant thereof means any connection or coupling, either direct or indirect, between two or more elements; the coupling or connection between the elements can be physical, logical, or a combination thereof. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word “or,” in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

The phrases “in some embodiments,” “according to some embodiments,” “in the embodiments shown,” “in other embodiments,” “in an implementation,” “in some implementations,” and the like generally mean the particular feature, structure, or characteristic following the phrase is included in at least one implementation of the present technology, and may be included in more than one implementation. In addition, such phrases do not necessarily refer to the same embodiments or different embodiments.

The above Detailed Description of examples of the technology is not intended to be exhaustive or to limit the technology to the precise form disclosed above. While specific examples for the technology are described above for illustrative purposes, various equivalent modifications are possible within the scope of the technology, as those skilled in the relevant art will recognize. For example, while processes or blocks are presented in a given order, alternative implementations may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed or implemented in parallel, or may be performed at different times. Further any specific numbers noted herein are only examples: alternative implementations may employ differing values or ranges.

The teachings of the technology provided herein can be applied to other systems, not necessarily the system described above. The elements and acts of the various examples described above can be combined to provide further implementations of the technology. Some alternative implementations of the technology may include not only additional elements to those implementations noted above, but also may include fewer elements.

These and other changes can be made to the technology in light of the above Detailed Description. While the above description describes certain examples of the technology, and describes the best mode contemplated, no matter how detailed the above appears in text, the technology can be practiced in many ways. Details of the system may vary considerably in its specific implementation, while still being encompassed by the technology disclosed herein. As noted above, particular terminology used when describing certain features or aspects of the technology should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific examples disclosed in the specification, unless the above Detailed Description section explicitly defines such terms. Accordingly, the actual scope of the technology encompasses not only the disclosed examples, but also all equivalent ways of practicing or implementing the technology under the claims.

To reduce the number of claims, certain aspects of the technology are presented below in certain claim forms, but the applicant contemplates the various aspects of the technology in any number of claim forms. For example, while only one aspect of the technology is recited as a computer-readable medium claim, other aspects may likewise be embodied as a computer-readable medium claim, or in other forms, such as being embodied in a means-plus-function claim. Any claims intended to be treated under 35 U.S.C. § 112(f) will begin with the words “means for”, but use of the term “for” in any other context is not intended to invoke treatment under 35 U.S.C. § 112(f). Accordingly, the applicant reserves the right to pursue additional claims after filing this application to pursue such additional claim forms, in either this application or in a continuing application.