Power-management virtual machines

Modern central processing units (CPUs), or processors, support a variety of processor power management technologies that can be used to balance performance, power consumption, and heat generation, among other things.

For example, modern CPUs support a variety of processor idle states, which reduce CPU power consumption and heat generation when the CPU is idle (e.g., not actively executing instructions). For example, the Advanced Configuration and Power Interface (ACPI) specification defines a variety of processor idle states, known as “C-states” (sometimes referred to as “C-modes”), as well as interfaces that an operating system (OS) can use to instruct a CPU to enter various C-states. C-states are states when the CPU has reduced or turned off selected functions. Different processors support different numbers of C-states in which various parts of the CPU are turned off. Generally, a higher-numbered C-state is a “deeper” idle state that turns off more parts of the CPU, while a lower-numbered C-state is a “lighter” idle state that turns off fewer parts of the CPU. Deeper idle states can significantly reduce power consumption by the CPU, as compared to lighter idle states. While the C-states implemented by a given CPU may vary, some basic C-states defined by the ACPI specification, and supported by most contemporary processors, are outlined in Table 1:

Some processor manufacturers define additional C-states, which may vary by processor model. For example, contemporary processors from INTEL CORPORATION of Santa Clara, California, have C-states up to C10, where the processor distinguishes core states (e.g., states of individual CPU cores) and package states (e.g., groupings of CPU cores within the same processor package).

Many modern CPUs also support dynamic frequency scaling technologies, which adjust the clock frequency of a CPU depending on processing needs. The use of dynamic frequency scaling technologies can conserve power and reduce the amount of heat generated by the CPU. Dynamic frequency scaling technologies come in power-saving and performance-boosting forms. In power-saving forms, the processor operates at a base clock rate but can be throttled to lower clock rates (e.g., when power resources are constrained, during periods of reduced workload demands). Example power-saving technologies include SPEED STEP from INTEL, and COOL'N'QUIET and POWERNOW! from ADVANCED MICRO DEVICES, INC. (AMD) of Santa Clara, California. In performance-boosting forms, the processor operates at a base clock rate but can “boost” to higher clock rates (e.g., during periods of increased workload demands, and when thermal conditions permit). Examples include TURBO BOOST from INTEL and TURBO CORE from AMD.

Many modern CPUs also support overclocking, in which the clock rate of a CPU is increased to exceed a clock rate recommended or certified by the manufacturer. While overclocking can lead to improved performance, it comes with increased power consumption and heat generation, a risk of unstable behavior, and a shortened CPU lifespan.

Additionally, hypervisor-based virtualization technologies allocate portions of a computer system's physical resources (e.g., processor cores, physical memory regions, storage resources) into separate partitions and execute software within each partition. Hypervisor-based virtualization technologies, therefore, facilitate the creation of virtual machines (VMs) that each executes guest software, such as an OS and other software executing therein. A computer system hosting VMs is commonly called a VM host (sometimes called a “VM host node”). While hypervisor-based virtualization technologies can take a variety of forms, many use an architecture comprising a hypervisor that has direct access to hardware, and that operates in a separate execution environment from all other software in the system, a host partition that executes a host OS and a host virtualization stack, and one or more guest partitions corresponding to VMs. The host virtualization stack within the host partition manages guest partitions. Thus, the hypervisor grants the host partition a greater level of access to the hypervisor itself and to hardware resources than it does to guest partitions.

Virtualization service providers operate a plurality of VM hosts to provide VM hosting services to a plurality of tenants. In doing so, virtualization service providers may collocate VMs from a plurality of tenants at a single VM host. Examples of virtualization service providers include AZURE operated by MICROSOFT CORPORATION of Redmond, Washington; AMAZON WEB SERVICES (AWS) operated by AMAZON, INC. Seattle, Washington; and GOOGLE CLOUD PLATFORM (GCP) operated by GOOGLE LLC of Mountain View, California.

The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one example technology area where some embodiments described herein may be practiced.

In some aspects, the techniques described herein relate to methods, systems, and computer program products, including determining that a virtual machine (VM) operating at a computer system possesses a power management entitlement; identifying an architectural power management feature available at a physical processor core of a processor system, the physical processor core being associated with a virtual processor core of the VM; based on the VM possessing the power management entitlement, presenting an interface to the VM, the interface exposing the architectural power management feature to the VM; identifying a request from the VM to modify a state of the architectural power management feature; and based on the request, modifying the state of the architectural power management feature at the physical processor core.

In some aspects, the techniques described herein relate to methods, systems, and computer program products, including: determining that a VM operating at a computer system possesses a power management entitlement; identifying an architectural power management feature available at a physical processor core of a processor system, the physical processor core being associated with a virtual processor core of the VM; based on the VM possessing the power management entitlement, presenting a para-virtual service to the VM, the para-virtual service presenting an application programming interface (API) enabling a para-virtual driver operating at the VM to make API calls to manage the architectural power management feature; at the para-virtual service, identifying an API call from the para-virtual driver, the API call requesting a modification to a state of the architectural power management feature; and based on identifying the API call from the para-virtual driver, modifying the state of the architectural power management feature at the physical processor core.

In some aspects, the techniques described herein relate to methods, systems, and computer program products, including: determining that a VM operating at a computer system possesses a power management entitlement; identifying an architectural power management feature available at a physical processor core of a processor system, the physical processor core being associated with a virtual processor core of the VM; based on the VM possessing the power management entitlement, presenting an interface to the VM, the interface exposing the architectural power management feature to the VM as an architectural feature of the virtual processor core; identifying a modification by the VM to a state of the architectural feature of the virtual processor core; and based on the modification to the state of the architectural feature of the virtual processor core, modifying a state of the architectural power management feature at the physical processor core.

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Hypervisors generally prohibit virtual machines (VMs) from configuring power management technologies (e.g., processor idle states, dynamic frequency scaling, overclocking) at VM host processors. This prevents VM host processors from being placed by a VM into unknown or undesirable power and/or performance states, and protects VM host processors from potential instability, excessive heat generation, and/or damage (e.g., due to overclocking). Additionally, for virtualization host providers, this ensures that the virtualization host provider retains control over managing a balance between VM power consumption and VM performance.

Despite these advantages of prohibiting VMs from configuring power management technologies at VM host processors, it is often desirable for virtualization host providers to let VMs do so under regulated conditions. Accordingly, at least some embodiments described herein expose power management technologies to VMs possessing an entitlement (e.g., a power management entitlement) and enable those VMs to modify processor-based settings relating to those power management technologies at VM host processors (e.g., the VM host processor(s) to which a VM's virtual processor(s) are associated).

Enabling VMs to configure power management technologies at VM host processors under regulated conditions provides a number of advantages. One advantage is that a VM operator may choose to reduce energy consumption by the VM by enabling power-saving features at the VM. Another advantage is that a VM operator may be able to boost performance at a virtual processor under the tenant's control by enabling power-saving features at another virtual processor under the tenant's control. For instance, if the virtual processors are associated with physical processors that are thermally linked (e.g., part of the same processor package), enabling power-saving features at one physical processor may reduce heat generation by that physical processor and enable the other virtual processor to better utilize dynamic frequency scaling technologies (e.g., TURBO BOOST). Another advantage is that the VM operator may be able to ensure consistent performance at a virtual processor and/or limit compute usage by the virtual processor, by disabling dynamic frequency scaling technologies (e.g., TURBO BOOST) at an associated physical processor.

illustrates an exampleof a computer architecture that facilitates managing a processor power management feature on behalf of a VM. In example, the computer architecture includes a computer systemcomprising hardware. An illustrated example of hardwareincludes a processor system. In embodiments, processor systemis a single central processing unit (CPU) comprising one or more processors (e.g., CPU cores) or a plurality of CPUs which each includes one or more cores. Regardless of the arrangement of processor system, in, processor systemincludes a plurality of CPU cores, illustrated as physical cores(e.g., physical coreto physical core, with an ellipsis indicating that there could be any number of physical cores). For example, each of physical coresis a single-threaded physical core or is one thread of a simultaneous multithreading (SMT) physical core.

Illustrated examples of hardwarealso include a memory(e.g., system or main memory), a storage media(e.g., a single computer-readable storage medium or a plurality of computer-readable storage media), and a network interface(e.g., one or more network interface cards). Although not shown, other examples of hardwareinclude a trusted platform module (TPM) for facilitating measured boot features, an input/output (I/O) memory management unit (IOMMU) that connects a direct memory access (DMA)-capable I/O bus (and any devices connected thereto) to memory, a graphics processing unit (GPU) for rendering image data, a video display interface for connecting to display hardware, a user input interface for connecting to user input devices, an external bus for connecting to external devices, and the like.

As illustrated in example, a hypervisorexecutes directly on hardware. In general, hypervisorpartitions hardware resources (e.g., processor system, memory, I/O resources) into a plurality of partitions. In embodiments, these partitions include a host partitionwithin which a host OS (not illustrated) executes. In embodiments, these partitions also include guest partitionswithin which guest OSs execute (e.g., guest partitionexecuting guest OSto guest partitionexecuting guest OS, with an ellipsis indicating that hypervisorcould operate any number of guest partitions).

As illustrated, host partitionincludes a virtualization stack, which uses application program interface (API) calls (e.g., hypercalls) to hypervisorto create, manage, and destroy guest partitions. In embodiments, virtualization stackmakes decisions about which portion(s) of memoryto allocate to each guest partition, operates para-virtual drivers that multiplex guest partition access to physical hardware devices (e.g., storage media, network interface), and facilities limited communications among partitions via a VM bus, among other things.

In embodiments, hypervisorcreates one or more virtual processors for each partition. For example,illustrates host partitionas including virtual cores(e.g., virtual coreto virtual core, with an ellipsis indicating that there could be any number of virtual cores), illustrates guest partitionas including virtual cores(e.g., virtual coreto virtual core, with an ellipsis indicating that there could be any number of virtual cores), and illustrates guest partitionas including virtual cores(e.g., virtual coreto virtual core, with an ellipsis indicating that there could be any number of virtual cores). In embodiments, hypervisoruses a processor managerto manage the use of processor system(e.g., physical cores) by these virtual processors. In embodiments, hypervisorcoordinates with virtualization stackin managing the use of processor systemby virtual processors. Thus, in, processor manageris illustrated as being split between hypervisor(processor manager) and virtualization stack(processor manager) to indicate that, in some examples, the functionality of processor manageris split among virtualization stackand hypervisor.

In embodiments, hypervisoralso allocates a portion of memoryto each partition and intercepts and routes any interrupts generated by each partition, among other things. In embodiments, hypervisoruses second-level address translation (SLAT) to isolate memory allocated to each partition created by hypervisorfrom other partition(s) created by hypervisor. For example, hypervisormay use one or more SLAT tables to map system physical addresses (SPAs) in memoryto guest physical addresses (GPAs) that make up each partition's memory space.

In embodiments, each physical core of physical coressupports one or more power management technologies, such as idle states, dynamic frequency scaling, overclocking, and the like. In some embodiments, the configuration of these power management technologies is controlled on a per-package basis, a per-core basis, etc. In one example, power management technologies are configured for a physical core by writing to a processor register at that core. In, a configuration of one or more power management technologies at each physical core of physical coresis denoted by power settings(e.g., power settingfor physical coreto power settingfor physical core).

In embodiments, any guest partition of guest partitionscan be granted a power management entitlement, such as an entitlement as being a “power management” VM that is enabled to see and modify one or more power management technologies at physical cores. In, an enlightenment componentstores or otherwise accesses (e.g., from virtualization stack) a set of enlightenments that specify which guest partition(s) possess a power management entitlement. In embodiments, enlightenments are granted or otherwise configured by an administrator (e.g., a virtualization service provider) of computer system. In, enlightenment componenthas determined that guest partitionpossesses a power management entitlement, and thus guest partitionis indicated as being a power management VM. In contrast, enlightenment componenthas determined that guest partitionlacks a performance entitlement, and thus guest partitionis indicated as being a conventional VM.

In embodiments, processor managerexposes power management technologies of one or more physical cores of physical coresto one or more guest partitions (e.g., guest partition) of guest partitionsthat possess a power management entitlement. In embodiments, processor managerenables guest partition(s) with this entitlement to read and modify power settingsrelating to those power management technologies.

In, processor manageris illustrated as including a power manager, while guest partitionis illustrated as including a power client. In embodiments, power manageridentifies power management features available at physical cores, such as idle state features, frequency scaling features, overclocking features, and the like. In embodiments, power managerexposes these power management features, including their settings, to power client. Correspondingly, in embodiments, power clientreceives power management features and their settings from power managerand communicates requests to modify those settings to power manager. The forms of power managerand power clientcan vary, depending on implementation. Examples are now described in connection withto.

Turning to, illustrated is an exampleof a computer architecture that exposes power management architectural interfaces to a VM via a virtual processor. In example, power manageris an architectural interfacethat exposes one or more power management features of a physical core via a virtual core. Thus, in these embodiments, the virtual core is the power client. For instance, in example, architectural interfaceexposes power management feature(s) of physical coreto guest partitionvia virtual core. In embodiments, a CPU control componentof guest OSreads power settingfor physical corevia virtual core(e.g., by reading from a processor register at virtual core) and modifies power settingvia interactions with virtual core(e.g., by writing to a processor register at virtual core).

In some embodiments, architectural interfaceexposes all power management features of a physical core. In other embodiments, architectural interfaceexposes a subset of power management features of a physical core, thereby limiting a power management VM's ability to interact with the physical core's management features. Thus, in some embodiments, architectural interfacefilters which power management features are available to a power management VM.

It is noted that, in embodiments, because CPU control componentat guest OScan directly configure power setting(via interacting with virtual core), it may be possible for guest partitionto place power settinginto a state that is not reversible by hypervisorwithout a reset of physical core

Turning to, illustrated is an exampleof a computer architecture that exposes power management architectural interfaces to a VM via an API. In one embodiment, this API is presented by a para-virtual service. In general, para-virtual services present software interface VMs that operate corresponding para-virtual drivers. Such VMs are thus “enlightened” to be aware that they are operating in a virtualized environment. In example, power manageris a para-virtual servicethat exposes one or more power management features of a physical core via a set of APIs, and power clientis a para-virtual driverthat interacts with the APIs of para-virtual service. For instance, in example, para-virtual serviceexposes power management feature(s) of physical coreto para-virtual driveroperating at guest partition. In embodiments, para-virtual driverinteracts with para-virtual servicevia calls (e.g., hypercalls) to a set of APIs presented by para-virtual service.

In embodiments, para-virtual drivermakes API call(s) requesting the power management feature(s) available at physical core, and para-virtual servicereturns those power management feature(s). In embodiments, para-virtual drivermakes API call(s) requesting the parameter(s) available for a given power management feature, and para-virtual servicereturns those parameter(s). In embodiments, para-virtual drivermakes API call(s) requesting a modification to a power management feature, and para-virtual serviceenacts that change at power setting

Notably, in embodiments of example, hypervisorremains in control of changes to power settings. Thus, for example, hypervisorcan prevent certain changes from being made by a power management VM. In some embodiments, para-virtual servicetracks changes made by power management VMs and can use this tracked data to revert changes made by power management VMs. For example, para-virtual servicetracks modifications to power settingmade based on requests from para-virtual driverand thus can revert those modifications (e.g., in response to physical corebeing disassociated from power setting).

In some embodiments, para-virtual serviceexposes all power management features of a physical core. In other embodiments, para-virtual serviceexposes a subset of power management features of a physical core, thereby limiting a power management VM's ability to interact with the physical core's management features. Thus, in some embodiments, para-virtual servicefilters which power management features are available to a power management VM.

Embodiments are now described in connection with, which illustrates a flow chart of an example methodfor managing a processor power management feature on behalf of a VM. In embodiments, instructions for implementing methodare encoded as computer-executable instructions (e.g., representing processor manager) stored on a computer storage media (e.g., storage media) that are executable by a processor system (e.g., processor system) to cause a computer system (e.g., computer system) to perform method.

The following discussion now refers to a number of methods and method acts. Although the method acts may be discussed in certain orders or may be illustrated in a flow chart as occurring in a particular order, no particular ordering is required unless specifically stated, or required because an act is dependent on another act being completed prior to the act being performed.

Referring to, in embodiments, methodcomprises an actof determining that a VM possesses a power management entitlement. In some embodiments, actcomprises determining that a VM operating at the computer system possesses a power management entitlement. In an example, using enlightenment component, processor managerdetermines that guest partitionpossesses a power management entitlement. Thus, guest partitioncorresponds to a power management VM.

Methodalso comprises an actof identifying an architectural power management feature of a physical processor. In some embodiments, actcomprises identifying an architectural power management feature available at a physical processor core of the processor system, the physical processor core being associated with a virtual processor core of the VM. In an example, based on an association between virtual coreand physical core, power manageridentifies which power management feature(s) are available at physical core

In embodiments, the architectural power management feature is one of an idle state feature, a frequency scaling feature, or an overclocking feature. In some embodiments, identifying the architectural power management feature of the physical processor comprises identifying a plurality of architectural power management features. In these embodiments, the plurality of architectural power management features includes a combination of an idle state feature, a frequency scaling feature, and/or an overclocking feature.

Notably, in, there is no ordering specified between actand act. Thus, in embodiments, actand actcould be performed serially (in either order) or at least partially in parallel.

After actand act, methodalso comprises an actof presenting an interface exposing the architectural power management feature to the VM. In some embodiments, actcomprises, based on the VM possessing the power management entitlement, presenting an interface to the VM, the interface exposing the architectural power management feature to the VM. In an example, processor managerpresents power managerto guest partition. Power managerexposes available power management features at physical coreto guest partitionand enables guest partitionto modify power settingrelating to those power management features.

As described in connection withto, the nature of power managercan vary depending on implementation. Examples include architectural interfaceand para-virtual service.

As demonstrated in example, in some embodiments, presenting the interface to the VM comprises presenting the architectural power management feature to the VM as an architectural feature of the virtual processor core. For instance, in example, architectural interfaceexposes a power management feature to guest partitionvia virtual core. Thus, in some embodiments, actcomprises, based on the VM possessing the power management entitlement, presenting an interface to the VM, the interface exposing the architectural power management feature to the VM as an architectural feature of the virtual processor core.

As demonstrated in example, in some embodiments, presenting the interface to the VM comprises presenting a para-virtual service to the VM. For instance, in example, processor managerexposes para-virtual serviceto guest partition, which exposes a set of APIs to para-virtual driveroperating at guest OS. Thus, in some embodiments, actcomprises, based on the VM possessing the power management entitlement, presenting a para-virtual service to the VM, the para-virtual service presenting an API enabling a para-virtual driver operating at the VM to make API calls to manage the architectural power management feature.

In embodiments, a set of APIs presented by para-virtual serviceenable para-virtual driverto discover available power management features, and their parameters, and to request modifications. Thus, in embodiments of act, presenting the para-virtual service to the VM comprises exposing an API to the VM, the API enabling a para-virtual driver operating at the VM to discover the availability of the architectural power management feature. Additionally, in embodiments of act, presenting the para-virtual service to the VM comprises exposing an API to the VM, the API enabling a para-virtual driver operating at the VM to discover a parameter of the architectural power management feature. Additionally, in embodiments of act, presenting the para-virtual service to the VM comprises exposing an API to the VM, the API enabling a para-virtual driver operating at the VM to request a modification to a state of the architectural power management feature (e.g., changing a parameter of the architectural power management feature).

Methodalso comprises an actof identifying a request to modify the architectural power management feature. In some embodiments, actcomprises identifying a request from the VM to modify a state of the architectural power management feature. For example, power managerreceives a request from power clientto modify a state of a power management feature. As described in connection withto, the nature of this request can vary depending on implementation.

As demonstrated in example, in some embodiments identifying the request from the VM to modify the state of the architectural power management feature comprises identifying a modification by the VM to a state of the architectural feature of the virtual processor core. For instance, in example, architectural interfaceidentifies a write to a register at virtual core. Thus, in some embodiments, actcomprises identifying a modification by the VM to a state of the architectural feature of the virtual processor core.

As demonstrated in example, in some embodiments identifying the request from the VM to modify the state of the architectural power management feature comprises identifying an API call from the para-virtual driver to the para-virtual service. For instance, in example, para-virtual serviceidentifies an API request (e.g., a hypercall) from para-virtual driverthat requests a modification to power setting. Thus, in some embodiments, actcomprises, at the para-virtual service, identifying an API call from the para-virtual driver, the API call requesting a modification to a state of the architectural power management feature.

Methodalso comprises an actof modifying the architectural power management feature at a physical processor. In some embodiments, actcomprises, based on the request, modifying the state of the architectural power management feature at the physical processor core. In an example, power managerenacts the requested modification at power settingof physical core. Thus, via power manager, guest partitionhas modified a setting for a power management feature at physical core

Referring to example, in embodiments when actcomprises architectural interface, identifying a modification by the VM to the state of the architectural feature of the virtual processor core, actcomprises, based on the modification to the state of the architectural feature of the virtual processor core, modifying the state of the architectural power management feature at the physical processor core. For example, when CPU control componentwrites to a register at virtual core, architectural interfacewrites to a corresponding register at physical core

Referring to example, in embodiments when actcomprises the para-virtual service identifying an API call from the para-virtual driver that requests a modification to the state of the architectural power management feature, actcomprises, based on identifying the API call from the para-virtual driver, modifying the state of the architectural power management feature at the physical processor core. For example, when para-virtual servicereceives an API request from para-virtual driverto modify a setting for a power management feature, para-virtual serviceenacts that modification at power setting

In some embodiments, para-virtual servicedoes not track the modification to the state of the architectural power management feature at the physical processor core, and thus, para-virtual servicemay be unable to revert the modification absent a processor restart. In other embodiments, para-virtual servicetracks the modification to the state of the architectural power management feature at the physical processor core. In these embodiments, para-virtual servicemay later revert the modification to the state of the architectural power management feature at the physical processor core (e.g., based on the disassociation of the virtual processor core from the physical processor core).