Dynamically Adjusting Performance Tuning Parameters in a Network System

This application is a continuation of U.S. patent application Ser. No. 17/455,867, filed 19 Nov. 2021, the entire contents of which is incorporated herein by reference.

This disclosure relates to computing systems, and more specifically to configuring parameters of network nodes in a network system.

In a typical cloud data center environment, there is a large collection of interconnected servers that provide computing and/or storage capacity to run various applications. For example, a data center may comprise a facility that hosts applications and services for subscribers, i.e., customers of data center. The data center may, for example, host all of the infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via high-speed switch fabric provided by one or more tiers of physical network switches and routers. More sophisticated data centers provide infrastructure spread throughout the world with subscriber support equipment located in various physical hosting facilities.

Virtualized data centers are becoming a core foundation of the modern information technology (IT) infrastructure. In particular, modern data centers have extensively utilized virtualized environments in which virtual hosts, also referred to herein as virtual execution elements, such virtual machines or containers, are deployed and executed on an underlying compute platform of physical computing devices.

Virtualization within a data center or any environment that includes one or more servers can provide several advantages. One advantage is that virtualization can provide significant improvements to efficiency. As the underlying physical computing devices (i.e., servers) have become increasingly powerful with the advent of multicore processor architectures with a large number of cores per physical central processing unit (CPU), virtualization becomes easier and more efficient. A second advantage is that virtualization provides significant control over the computing infrastructure. As physical computing resources become fungible resources, such as in a cloud-based computing environment, provisioning and management of the computing infrastructure becomes easier. Thus, enterprise IT staff often prefer virtualized compute clusters in data centers for their management advantages in addition to the efficiency and increased return on investment (ROI) that virtualization provides.

Containerization is a virtualization scheme based on operating system-level virtualization. Containers are light-weight and portable execution elements for applications that are isolated from one another and from the host. Because containers are not tightly-coupled to the host hardware computing environment, an application can be tied to a container image and executed as a single light-weight package on any host or virtual host that supports the underlying container architecture. As such, containers address the problem of how to make software work in different computing environments. Containers offer the promise of running consistently from one computing environment to another, virtual or physical.

With containers' inherently lightweight nature, a single host can often support many more container instances than traditional virtual machines (VMs). Often short-lived, containers can be created and moved more efficiently than VMs, and they can also be managed as groups of logically-related elements (sometimes referred to as “pods” for some orchestration platforms, e.g., Kubernetes). A network system that manages deployment and infrastructure for application execution may use pods as part of orchestration, e.g., for automating deployment, scaling, and operations of applications across clusters of hosts and providing computing infrastructure, which may include container-centric computing infrastructure. A network system may also use pods for network management, e.g., for creating virtual networks in the network infrastructure to enable packetized communication among applications running on virtual execution environments, such as containers or VMs, as well as among applications running on legacy (e.g., physical) environments.

In general, the disclosure describes a performance controller for a container-based orchestration system that implements techniques that can determine performance parameters based on node configurations and can dynamically adjust the parameters as conditions and workloads in the network system change. In large scale deployment environments such as data centers, there can be multinode clusters where the nodes in the cluster may have different configurations and different available resources. Further, the nodes may be used by different applications having different runtime characteristics and requirements. For example, some applications may have large memory requirements while other applications require fast response time. Various parameters can affect the performance aspects of nodes and applications running on the nodes. The values of these parameters may be different depending on the resources (real and virtual) available on the node and workloads hosted by the node. The process of setting or adjusting parameters on a node to attempt to achieve a performance characteristic or for a particular type of workload, application, or environment can be referred to as “tuning” the node. Similarly, the parameters that may be adjusted to tune a node to achieve a performance characteristic or for a particular type of workload, application or environment may be referred to as “tuning parameters.”

As an example, a node that provides virtual routing services may provide such services with a virtual router that runs as a user mode application in user space memory of the node, as a kernel mode vrouter that runs in kernel space memory, or a vrouter that is integrated with a smart network interface card (NIC). Nodes that host virtual routers that run in user memory space may have different tuning parameter values from nodes that host a kernel mode virtual router that executes in kernel memory space. In some cases, tuning a node for a particular performance characteristic may cause a decrease in another performance characteristic. For example, tuning a node for power savings may increase the performance of the node with respect to power usage, but may decrease the performance of the node with respect to CPU processing.

Existing systems do not provide the ability to dynamically and automatically reconfigure nodes or adjust tuning parameters to maximize performance for current workload conditions. If tuning is done at all, it is typically done manually (e.g., by an administrator) on a node by node basis, and once done, tuning parameters typically remain static and do not change. Because tuning the numerous nodes that may exist in a data center is tedious and error prone, often no tuning is done at all.

The disclosure describes techniques that dynamically and automatically reconfigure nodes and/or adjust parameters to improve or maximize performance for current workload conditions. A performance controller on a master node of a node cluster can communicate with performance agents on worker nodes of the cluster. The performance agent running on the nodes can both report current node configuration data to the performance controller and can receive node tuning parameters that may be based on the current node configuration and current workloads executing on the node.

The techniques disclosed herein may be included in a practical application that provides technical advantages over existing systems. For example, as noted above, existing systems do not dynamically and automatically reconfigure nodes or adjust tuning parameters to maximize performance for current workload conditions. A practical application of the techniques disclosed herein is a performance controller and performance agent that can set various parameters on a worker node to tune the worker node according to a node profile specifying desired characteristics for the worker node. A technical advantage of the techniques disclosed herein is that a system including a performance controller implementing the techniques can maximize node performance in a cluster using feedback received from performance agents executing on the worker nodes of the cluster. The performance controller can use the feedback received from a performance agent for a node to set values of various parameters to tune the node for maximum performance given the characteristics specified for by the node's assigned profile. The tuning parameter values can be provided to the performance agent by the performance controller, and the performance agent can enforce the tuning parameters for the node. Further, the tuning parameters may be adjusted over time as conditions within a node or cluster of nodes change. Thus, the techniques disclosed herein can be an improvement over existing systems by facilitating maximizing performance for clusters having large numbers of worker nodes.

In one example, a method includes receiving, by a master node of a cluster and from each agent executing on a corresponding worker node of a plurality of worker nodes of the cluster, a plurality of node characteristics associated with the corresponding worker node of the plurality of worker nodes, wherein the cluster is a container-based orchestration system cluster, and wherein the master node manages orchestration of workloads on the worker nodes of the cluster; for each worker node of the plurality of worker nodes, determining, by the master node, one or more parameters corresponding to the plurality of node characteristics associated with the corresponding worker node and a node profile of the corresponding worker node, and providing, by the master node to the corresponding agent executing on the worker node, the one or more parameters, wherein the corresponding agent configures the worker node in accordance with the one or more parameters; receiving, by the master node, a request to deploy a pod having one or more pod characteristics; selecting, by the master node and based on the plurality of node characteristics and the one or more pod characteristics, a selected worker node of the plurality of worker nodes on which to deploy the pod; and scheduling, by the master node, the pod on the selected worker node.

In another example, a worker node includes a plurality of processors; a memory coupled to the plurality of processors; an orchestration agent that deploys a pod to the worker node in response to instructions from a master node that orchestrates workloads on the worker node, the master node being in a same cluster of nodes as the worker node, wherein the cluster is a container-based orchestration system cluster; and a performance agent executable by the plurality of processors, the performance agent configured to: monitor resource usage on the worker node, report the resource usage to the master node, receive one or more parameters from the master node, receive an indication from the orchestration agent that the pod is deployed to the worker node, the worker node selected by the master node in accordance with the resource usage, and configure the worker node to execute workloads of the pod in accordance with the one or more parameters.

In another example, a container-based orchestration system includes a master node; and a plurality of worker nodes communicatively coupled to the master node, wherein the master node manages orchestration of workloads on each of the plurality of worker nodes, wherein the master node comprises memory storing instructions that, when executed by processing circuitry of the master node, causes the master node to: receive, from each agent executing on a corresponding worker node of the plurality of worker nodes of the cluster, a plurality of node characteristics associated with the corresponding worker node, for each worker node of the plurality of worker nodes, determine one or more parameters corresponding to the plurality of node characteristics associated with the corresponding worker node and a node profile of the corresponding worker node, and provide, to the corresponding agent executing on the worker node, the one or more parameters, wherein the corresponding agent configures the worker node in accordance with the one or more parameters; receive a request to deploy a pod having one or more pod characteristics, select a selected worker node of the plurality of worker nodes to receive the pod based on the plurality of node characteristics and the one or more pod characteristics, and schedule the pod on the selected worker node, and wherein the agent of the selected worker node is configured to: receive an indication that the pod is scheduled to the selected worker node, and configure the selected worker node to execute workloads of the pod in accordance with the one or more parameters.

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

is a conceptual diagram illustrating an example network that includes a performance controller for tuning performance parameters of nodes in the network.illustrates one example implementation of a network systemand a data centerthat hosts one or more computing networks, computing domains or projects, and/or cloud-based computing networks generally referred to herein as cloud computing cluster. The cloud-based computing clusters and may be co-located in a common overall computing environment, such as a single data center, or distributed across environments, such as across different data centers. Cloud-based computing clusters may, for example, be different cloud environments, such as various combinations of OpenStack cloud environments, Kubernetes cloud environments or other computing clusters, domains, networks, and the like. Other implementations of network systemand data centermay be appropriate in other instances. Such implementations may include a subset of the components included in the example ofand/or may include additional components not shown in.

In the example of, data centerprovides an operating environment for applications and services for customerscoupled to data centerby service provider network. Although functions and operations described in connection with network systemofmay be illustrated as being distributed across multiple devices in, in other examples, the features and techniques attributed to one or more devices inmay be performed internally, by local components of one or more of such devices. Similarly, one or more of such devices may include certain components and perform various techniques that may otherwise be attributed in the description herein to one or more other devices. Further, certain operations, techniques, features, and/or functions may be described in connection withor otherwise as performed by specific components, devices, and/or modules. In other examples, such operations, techniques, features, and/or functions may be performed by other components, devices, or modules. Accordingly, some operations, techniques, features, and/or functions attributed to one or more components, devices, or modules may be attributed to other components, devices, and/or modules, even if not specifically described herein in such a manner.

Data centerhosts infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. Service provider networkmay be coupled to one or more networks administered by other providers, and may thus form part of a large-scale public network infrastructure, e.g., the Internet.

In some examples, data centermay represent one of many geographically distributed network data centers. As illustrated in the example of, data centeris a facility that provides network services for customers. Customersmay be collective entities such as enterprises and governments or individuals. For example, a network data center may host web services for several enterprises and end users. Other example services may include data storage, virtual private networks, traffic engineering, file service, data mining, scientific- or super-computing, and so on. In some examples, data centeris an individual network server, a network peer, or otherwise.

In the example of, data centerincludes a set of storage systems, application servers, compute nodes, or other devices, including network devices. Network devicesmay include master nodeand worker nodeA through worker nodeN (collectively “worker nodes,” representing any number of worker nodes). Devicesmay be interconnected via high-speed switch fabricprovided by one or more tiers of physical network switches and routers. In some examples, devicesmay be included within fabric, but are shown separately for ease of illustration.

Network devicesmay be any of a number of different types of network devices (core switches, spine network devices, leaf network devices, edge network devices, or other network devices), but in some examples, one or more devicesmay serve as physical compute nodes or storage nodes of the data center. For example, one or more of devicesmay provide an operating environment for execution of one or more customer-specific applications or services. Alternatively, or in addition, one or more of devicesmay provide an operating environment for one or more virtual machines or other virtualized instances, such as containers. In some examples, one or more of devicesmay be alternatively referred to as a host computing device or, more simply, as a host. A network devicemay thereby execute one or more virtualized instances, such as virtual machines, containers, or other virtual execution environment for running one or more services, such as virtualized network functions (VNFs).

In general, each of network devicesmay be any type of device that may operate on a network and which may generate data (e.g., connectivity data, flow data, sFlow data) accessible through telemetry or otherwise, which may include any type of computing device, sensor, camera, node, surveillance device, or other device. Further, some or all of network devicesmay represent a component of another device, where such a component may generate data collectible through telemetry or otherwise. For example, some or all of network devicesmay represent physical or virtual network devices, such as switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection, and/or intrusion prevention devices.

Although not specifically shown, switch fabricmay include top-of-rack (TOR) switches coupled to a distribution layer of chassis switches, and data centermay include one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection, and/or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. Switch fabricmay perform layer 3 routing to route network traffic between data centerand customersby service provider network. Gatewayacts to forward and receive packets between switch fabricand service provider network.

Any network device of network devicesmay be configured with virtual execution elements by virtualizing resources of the server to provide an isolation among one or more processes (applications) executing on the server. “Hypervisor-based” or “hardware-level” or “platform” virtualization refers to the creation of virtual machines that each includes a guest operating system for executing one or more processes. In general, a virtual machine provides a virtualized/guest operating system for executing applications in an isolated virtual environment. Because a virtual machine is virtualized from physical hardware of the host server, executing applications are isolated from both the hardware of the host and other virtual machines. Each virtual machine may be configured with one or more virtual network interfaces for communicating on corresponding virtual networks.

Virtual networks are logical constructs implemented on top of the physical networks. Virtual networks may be used to replace VLAN-based isolation and provide multi-tenancy in a virtualized data center, e.g., data center. Each tenant or an application can have one or more virtual networks. Each virtual network may be isolated from all the other virtual networks unless explicitly allowed by security policy.

Virtual networks can be connected to, and extended across physical Multi-Protocol Label Switching (MPLS) Layer 3 Virtual Private Networks (L3VPNs) and Ethernet Virtual Private Networks (EVPNs) networks using a data centeredge router (not shown in). Virtual networks may also be used to implement Network Function Virtualization (NFV) and service chaining.

Virtual networks can be implemented using a variety of mechanisms. For example, each virtual network could be implemented as a Virtual Local Area Network (VLAN), Virtual Private Networks (VPN), etc. A virtual network can also be implemented using two networks—the physical underlay network made up of switch fabricand a virtual overlay network. The role of the physical underlay network is to provide an “IP fabric,” which provides unicast Internet protocol (IP) connectivity from any physical device (server, storage device, router, or switch) to any other physical device. The underlay network may provide uniform low-latency, non-blocking, high-bandwidth connectivity from any point in the network to any other point in the network.

As described further below with respect to virtual router (vrouter)A, virtual routers running in the network devicescreate a virtual overlay network on top of the physical underlay network using a mesh of dynamic “tunnels” amongst themselves. These overlay tunnels can be MPLS over GRE/UDP tunnels, or VXLAN tunnels, or NVGRE tunnels, for instance. Virtual routers may also support segment routing-MPLS (SR-MPLS) and IPV6 underlays. The underlay physical routers and switches may not contain any per-tenant state for virtual machines or other virtual execution elements, such as any Media Access Control (MAC) addresses, IP address, or policies. The forwarding tables of the underlay physical routers and switches may, for example, only contain the IP prefixes or MAC addresses of the network devices. (Gateway routers or switches that connect a virtual network to a physical network are an exception and may contain tenant MAC or IP addresses.)

“Container-based” or “operating system” virtualization refers to the virtualization of an operating system to run multiple isolated systems on a single machine (virtual or physical). Such isolated systems represent containers, such as those provided by Kubernetes Container Runtime Interface-Open Container Initiative (CRI-O), containerd, the open-source Docker Container application or by CoreOS Rkt (“Rocket”). Like a virtual machine, each container is virtualized and may remain isolated from the host machine and other containers. However, unlike a virtual machine, each container may omit an individual operating system and provide only an application suite and application-specific libraries. In general, a container is executed by the host machine (e.g., one of network devices) as an isolated user-space instance and may share an operating system and common libraries with other containers executing on the host machine.

Thus, containers may require less processing power, storage, and network resources than virtual machines. A group of one or more containers may be configured to share one or more virtual network interfaces for communicating on corresponding virtual networks. In some examples, containers are managed by their host kernel to allow limitation and prioritization of resources (CPU, memory, block I/O, network, etc.) without the need for starting any virtual machines. In some examples, containers may be deployed according to Linux Containers (LXC), an operating-system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

Network deviceshost virtual network endpoints for one or more virtual networks that operate over the physical network represented inby switch fabric. Although described primarily with respect to a data center-based switching network, other physical networks, such as service provider network, may underlay the one or more virtual networks.

In some examples network systemincludes orchestrator, and SDN controller. Virtual execution elements may be deployed to a virtualization environment using a cluster-based framework in which a cluster master node (e.g., master node) of a cluster manages the deployment and operation of containers to one or more cluster worker nodes (e.g., worker nodesA-N) of the cluster. The terms “master node” and “worker node” used herein encompass different orchestration platform terms for analogous devices that distinguish between primarily management elements of a cluster and primarily virtual execution element hosting devices of a cluster.

Orchestratorand SDN controllermay execute on separate computing devices, or may execute on the same computing device. Each of orchestratorand SDN controllermay be a distributed application that executes on one or more computing devices. Orchestratorand SDN controllermay implement respective master nodes for one or more clusters each having one or more worker nodesimplemented by respective network devices. In general, SDN controllercontrols the network configuration of the data centerfabric to, e.g., establish one or more virtual networks for packetized communications among virtual network endpoints. SDN controllerprovides a logically and in some cases physically centralized controller for facilitating operation of one or more virtual networks within data center. In some examples, SDN controllermay operate in response to configuration input received from orchestratorand/or an administrator/operator. Additional information regarding SDN controlleroperating in conjunction with other devices of data centeror other software-defined network is found in International Application Number PCT/US2013/044378, filed Jun. 5, 2013, and entitled “PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET FLOWS;” in U.S. Pat. No. 9,571,394, issued Feb. 14, 2017, and entitled “Tunneled Packet Aggregation for Virtual Networks;” and in U.S. Pat. No. 10,728,145, issued Jul. 28, 2020 and entitled “MULTIPLE VIRTUAL NETWORK INTERFACE SUPPORT FOR VIRTUAL EXECUTION ELEMENTS,” each which is incorporated by reference as if fully set forth herein.

In general, orchestratorcontrols the deployment, scaling, and operations of virtual execution elements across clusters of network devicesand providing computing infrastructure, which may include container-centric computing infrastructure. In some examples, orchestratormanages functions of data centersuch as compute, storage, networking, and application resources. For example, orchestratormay create a virtual network for a tenant within data centeror across data centers. Orchestratormay attach virtual machines (VMs) to a tenant's virtual network. Orchestratormay connect a tenant's virtual network to an external network, e.g., the Internet or a VPN. Orchestratormay implement a security policy across a group of VMs or to the boundary of a tenant's network. Orchestratormay deploy a network service (e.g., a load balancer) in a tenant's virtual network. Orchestratorand, in some cases, SDN controllermay implement respective cluster masters for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, each of which may provide virtualization infrastructure to the container management platform. Kubernetes is available from the Cloud Native Computing Foundation of San Francisco, California. Each of network devicesmay host one or more virtual execution elements each having at least one virtual network endpoint for one or more virtual networks configured in the physical network. A virtual network endpoint for a virtual network may represent one or more virtual execution elements that share a virtual network interface for the virtual network. For example, a virtual network endpoint may be a virtual machine, a set of one or more containers (e.g., a pod), or another other virtual execution element(s), such as a layer 3 endpoint for a virtual network. The term “virtual execution element” encompasses virtual machines, containers, and other virtualized computing resources that provide an at least partially independent execution environment for applications. The term “virtual execution element” may also encompass a pod of one or more containers. As shown in, worker nodeA hosts one virtual network endpoint in the form of podA having one or more containers. However, a worker nodemay execute as many virtual execution elements as is practical given hardware resource limitations of the worker node. Each of the virtual network endpoints may use one or more virtual network interfaces to perform packet I/O or otherwise process a packet. For example, a virtual network endpoint may use one virtual hardware component (e.g., a VNF) enabled by NICA to perform packet I/O and receive/send packets on one or more communication links with switch fabric. Other examples of virtual network interfaces are described below.

Worker nodeseach include at least one NIC, which each include at least one interface to exchange packets with switch fabricover a communication link. For example, worker nodeA includes NICA. Any of NICsmay provide one or more virtual hardware components such as a virtual routerfor virtualized input/output (I/O). A virtual hardware component for I/O may be a virtualization of a physical NIC(the “physical function”). For example, the physical function of the network interface card (or “network adapter”) is virtualized to present one or more virtual network interfaces as “virtual functions” for use by respective endpoints executing on the network device. In this way, the virtual network endpoints may share the same physical hardware resources and the virtual functions are examples of virtual hardware components. Virtual hardware components associated with NICA may be associated with a layer 2 destination address, which may be assigned by the NICA or a software process responsible for configuring NICA.

Virtual routersterminate virtual network overlay tunnels and determine virtual networks for received packets based on tunnel encapsulation headers for the packets, and forward packets to the appropriate destination virtual network endpoints for the packets. For worker nodeA, for example, for each of the packets outbound from virtual network endpoints hosted by worker nodeA (e.g., podA), the virtual routerA attaches a tunnel encapsulation header indicating the virtual network for the packet to generate an encapsulated or “tunnel” packet, and virtual routerA outputs the encapsulated packet via overlay tunnels for the virtual networks to a physical destination computing device, such as another one of network devices. As used herein, a virtual routermay execute the operations of a tunnel endpoint to encapsulate inner packets sourced by virtual network endpoints to generate tunnel packets and decapsulate tunnel packets to obtain inner packets for routing to other virtual network endpoints.

Network systemimplements an automation platform for automating deployment, scaling, and operations of virtual execution elements across network devicesto provide virtualized infrastructure for executing application workloads and services. In some examples, the platform may be a container orchestration platform that provides a container-centric infrastructure for automating deployment, scaling, and operations of containers to provide a container-centric infrastructure. “Orchestration,” in the context of a virtualized computing infrastructure generally refers to provisioning, scheduling, and managing virtual execution elements and/or applications and services executing on such virtual execution elements to the host servers available to the orchestration platform. Container orchestration, specifically, permits container coordination and refers to the deployment, management, scaling, and configuration, e.g., of containers to host servers by a container orchestration platform. Example instances of orchestration platforms include Kubernetes, Docker swarm, Mesos/Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

In one example, podA is a Kubernetes pod and an example of a virtual network endpoint. A pod is a group of one or more logically-related containers (not shown in), the shared storage for the containers, and options on how to run the containers. Where instantiated for execution, a pod may alternatively be referred to as a “pod replica.” Each container of podA is an example of a virtual execution element. Containers of a pod are co-located on a single server, co-scheduled, and run in a shared context. The shared context of a pod may be a set of Linux namespaces, cgroups, and other facets of isolation. Within the context of a pod, individual applications might have further sub-isolations applied. Pods such as podA may represent various workloads that are performed by network system.

Worker nodeA includes a container platformA for running containerized applications, such as those of podA. Container platformA receives requests from orchestratorto obtain and host, in worker nodeA, containers. Container platformA obtains and executes the containers.

Container platformA includes a network moduleA that configures virtual network interfaces for virtual network endpoints. The container platformA uses network moduleA to manage networking for pods, including podA. For example, the network moduleA creates virtual network interfaces to connect pods to virtual routerA and enable containers of such pods to communicate, via the virtual network interfaces, to other virtual network endpoints over the virtual networks. Network moduleA may, for example, insert a virtual network interface for a virtual network into the network namespace for containers of in podA and configure (or request to configure) the virtual network interface for the virtual network in virtual routerA such that the virtual routerA is configured to send packets received from the virtual network via the virtual network interface to containers of podA and to send packets received via the virtual network interface from containers of podA on the virtual network. Network moduleA may assign a network address (e.g., a virtual IP address for the virtual network) and may setup routes for the virtual network interface. In Kubernetes, by default all pods can communicate with all other pods without using network address translation (NAT).

As part of the process of creating podA, orchestratorsends a message to SDN controllerto request that SDN controllercreate respective virtual network interfaces for the multiple virtual networks (indicated in the configuration data). Orchestratormay store, send to, or otherwise notify SDN controllerof virtual network configuration objects for the multiple virtual networks specified for podA. For example, orchestratormay obtain a pod manifest that includes an annotation indicating an interface type for a virtual network for podA and deploy podA to a host computing device. In this example, orchestratormay store pod configuration data (e.g., virtual network configuration objects for the multiple virtual networks specified for podA) for podA. The pod configuration data may include the interface type for the virtual network for the pod. The pod configuration data may determine the interface type specified in the request to configure podA. SDN controllermay configure any virtual networks not already configured in the network system.

SDN controllerprocesses the pod creation request to generate interface configuration data for the multiple virtual network interfacesfor podA for communicating via virtual networks. The interface configuration data may include a container or pod unique identifier and a list or other data structure specifying, for each of virtual network interface, network configuration data for configuring the virtual network interface. Network configuration data for a virtual network interface may include a network name, assigned virtual network address, MAC address, and/or domain name server values. The multiple virtual network interfacesmay correspond to multiple virtual networks.

Network configuration data for podA may include data describing multiple different virtual network interfaces. For example, orchestratormay obtain a pod manifest that includes an annotation indicating an interface type for a virtual network for podA and deploy podA to a host computing device. In this example, orchestratormay store pod configuration data (e.g., the network configuration data for podA for a virtual network interfaceA) for podA. The pod configuration data may include the interface type for the virtual network for the pod. The pod configuration data may determine the interface type specified in the request to configure podA.

Performance controllerof master nodemay interact with performance agentsof worker nodes. Performance controllerand performance agentsmay implement one or more of the techniques described herein to adjust or tune parameters by which worker nodesoperate so as to improve or maximize the performance of worker nodeswith respect to various goals (e.g., low latency, maximum throughput, power savings, etc.). For example, performance controllerand performance agentsmay dynamically and automatically reconfigure worker nodesand/or adjust parameters of worker nodesto improve or maximize performance for current workload conditions. A performance agentexecuting on a worker nodecan both report current node configuration data to performance controllerand can receive node tuning parameters from performance controllerthat may be based on the current node configuration and current workloads executing on the node. Further details of the operation of performance controllerand performance agentare provided below with respect to.

is a block diagram illustrating aspects of a master node and worker nodes in a cluster of a network system in accordance with one or more aspects of the present disclosure. In the example illustrated in, clusterincludes a master nodeand worker nodesA-N (referred to generically as a “worker node”). A worker node may also be referred to as a “compute node.” Master nodemay be an example implementation of master nodeof, and worker nodesmay be example implementations of worker nodesof.

In some aspects, master nodeincludes controller manager, performance controller, API server, default scheduler, custom scheduler, and database. Performance controllermay be an example implementation of performance controllerof. Master nodemay manage worker nodesA-N in cluster. In some aspects, master nodeand worker nodesA-N can be Kubernetes nodes.

Databaseis a backing store for cluster data of cluster. Cluster data may include cluster state and configuration data. In some aspects, databasemay be implemented as a key value store. Databasemay be a central database or distributed database. In some aspects, databasecan be a Kubernetes etcd database. Databasecan store node profilesand pod specifications. In some aspects, a node profilecan be a Kubernetes configmap. Node profilecan include parameters that describe one or more nodes of worker nodesA-N. In some aspects, one or more of the parameters may be tuning parameters for a worker node. Examples of such parameters include Huge Pages, CPU Pinning, non-uniform memory access (NUMA) node binding, pod scheduling, CPU sharing restrictions, network interface card (NIC) configurations, CPU governors, operating system (OS) boot parameters, virtual router (vrouter) configurations, CPU yield parameters etc.

Huge Pages (also referred to as Super Pages or Large Pages) indicates that the worker node supports setting memory page sizes to a larger size than a default size (typically 4K bytes). Using a larger page size than the default page size can make memory management more efficient at the expense of potentially wasted memory.

CPU pinning determines how pods or applications are assigned to CPUs. An application or pod may be permanently assigned a fixed set of CPUs using CPU pinning. It can be advantageous to explicitly assign different processors to different components to competition for the same processor. For example, it can be advantageous from a performance perspective to pin a DPDK vrouter to processors that are different from the processors used by applications such as VNFs that use vrouter. In this example, the DPDK vrouter and processes using the DPDK vrouter will not have to compete for the same set of processors.