Cache Residency Control

Processors access data in memory to perform calculations. General purpose memory typically has very high latency compared to the processing speed of processor and thus caches are used to alleviate some of these problems. Caches are generally much smaller than system memory but have much better access characteristics (e.g., lower latency and higher bandwidth). However, because caches are smaller than memory, a great deal of effort is put into attempting to store data likely to be used in the future into caches. Mistakes made in such efforts can hinder performance. For example, cache space occupied by data that is never or rarely used is wasted space, especially if other data that is actually used is not in the cache. Thus, improvements to caching operations are constantly being made.

While large, the relatively poor access characteristics of system memory can act as a significant bottleneck to processor operations. Thus, computer systems typically include cache hierarchies that help to alleviate such issues. Caches employ a variety of techniques to store data that is considered likely to be used by the processor in the near future. This effectively hides the latency to system memory, as a “hit” (successful access) within the cache will return data to the processor faster than system memory can.

While beneficial, successfully operating a cache is a complex task. One consideration is regarding how to select data for “eviction” (removal) from the cache in the situation that the cache is “full” (or more particularly, in the situation that data is to be stored in the cache but where there are no empty slots in the cache that are appropriate for that data). A typical manner to do this is to utilize a cache replacement policy that tracks certain information about data in the cache (such as “age”) and evicts data based on such information. Such cache replacement policies are generally good at selecting data but are not perfect, and can select data for eviction that is needed again very soon, or can perform other errors.

Techniques are provided herein for allowing some level of program-directed control over cache replacement policies. Typically, cache replacement policies are controlled directly by the hardware with little to no room for direct intervention by software. The present disclosure allows software to specify priority hints for cache lines (units of data) in the cache. These priority hints can specify that a cache line is to have “high priority,” in which case such a cache line cannot be evicted and remains in the cache. A mechanism is also provided to eventually age out high priority cache lines, which allows for reclamation of the space used by such cache lines when space is needed. Additional details are techniques are provided as well.

1 FIG. 100 100 100 102 104 106 108 112 102 104 106 108 is a block diagram of an example computing devicein which one or more features of the disclosure can be implemented. In various examples, the computing deviceis one of, but is not limited to, for example, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, a tablet computer, or other computing device. The deviceincludes, without limitation, one or more processors, a memory, one or more auxiliary devices, and a storage. An interconnect, which can be a bus, a combination of buses, and/or any other communication component, communicatively links the one or more processors, the memory, the one or more auxiliary devices, and the storage.

102 104 102 104 102 104 In various alternatives, the one or more processorsinclude a central processing unit (CPU), a graphics processing unit (GPU), a CPU and GPU located on the same die, or one or more processor cores, wherein each processor core can be a CPU, a GPU, or a neural processor. In various alternatives, at least part of the memoryis located on the same die as one or more of the one or more processors, such as on the same chip or in an interposer arrangement, and/or at least part of the memoryis located separately from the one or more processors. The memoryincludes a volatile or non-volatile memory, for example, random access memory (RAM), dynamic RAM, or a cache.

108 106 114 114 114 The storageincludes a fixed or removable storage, for example, without limitation, a hard disk drive, a solid state drive, an optical disk, or a flash drive. The one or more auxiliary devicesinclude, without limitation, one or more auxiliary processors, and/or one or more input/output (“IO”) devices. The auxiliary processorsinclude, without limitation, a processing unit capable of executing instructions, such as a central processing unit, graphics processing unit, parallel processing unit capable of performing compute shader operations in a single-instruction-multiple-data form, multimedia accelerators such as video encoding or decoding accelerators, or any other processor. Any auxiliary processoris implementable as a programmable processor that executes instructions, a fixed function processor that processes data according to fixed hardware circuitry, a combination thereof, or any other type of processor.

106 116 116 116 102 116 116 116 102 The one or more auxiliary devicesincludes an accelerated processing device (“APD”). The APDmay be coupled to a display device, which, in some examples, is a physical display device or a simulated device that uses a remote display protocol to show output. The APDis configured to accept compute commands and/or graphics rendering commands from processor, to process those compute and graphics rendering commands, and, in some implementations, to provide pixel output to a display device for display. As described in further detail below, the APDincludes one or more parallel processing units configured to perform computations in accordance with, for example, a single-instruction-multiple-data (“SIMD”) or a single-instruction-multiple-thread (“SIMT”) paradigm. Thus, although various functionality is described herein as being performed by or in conjunction with the APD, in various alternatives, the functionality described as being performed by the APDis additionally or alternatively performed by other computing devices having similar capabilities that are not driven by a host processor (e.g., processor) and, optionally, configured to provide graphical output to a display device. For example, it is contemplated that any processing system that performs processing tasks in accordance with a SIMD paradigm may be configured to perform the functionality described herein. Alternatively, it is contemplated that computing systems that do not perform processing tasks in accordance with a SIMD paradigm perform the functionality described herein.

117 The one or more IO devicesinclude one or more input devices, such as a keyboard, a keypad, a touch screen, a touch pad, a detector, a microphone, an accelerometer, a gyroscope, a biometric scanner, or a network connection (e.g., a wireless local area network card for transmission and/or reception of wireless IEEE 802 signals), and/or one or more output devices such as a display device, a speaker, a printer, a haptic feedback device, one or more lights, an antenna, or a network connection (e.g., a wireless local area network card for transmission and/or reception of wireless IEEE 802 signals).

2 FIG. 100 116 102 104 102 120 122 126 102 116 120 102 122 116 126 102 116 122 138 116 is a block diagram of aspects of device, illustrating additional details related to execution of processing tasks on the APD. The processormaintains, in system memory, one or more control logic modules for execution by the processor. The control logic modules include an operating system, a kernel mode driver, and applications. These control logic modules control various features of the operation of the processorand the APD. For example, the operating systemdirectly communicates with hardware and provides an interface to the hardware for other software executing on the processor. The kernel mode drivercontrols operation of the APDby, for example, providing an application programming interface (“API”) to software (e.g., applications) executing on the processorto access various functionality of the APD. The kernel mode driveralso includes a just-in-time compiler that compiles programs for execution by processing components (such as the parallel processing unitsdiscussed in further detail below) of the APD.

116 116 118 102 116 102 The APDexecutes commands and programs for selected functions, such as graphics operations and non-graphics operations that are or can be suited for parallel processing. The APDcan be used for executing graphics pipeline operations such as pixel operations, geometric computations, and rendering an image to display devicebased on commands received from the processor. The APDalso executes compute processing operations that are not directly related to graphics operations, such as operations related to video, physics simulations, computational fluid dynamics, or other tasks, based on commands received from the processor.

116 132 138 102 138 138 The APDincludes compute unitsthat include one or more parallel processing unitthat perform operations at the request of the processorin a parallel manner according to a parallel processing paradigm, such as SIMD or SIMT. In such paradigms, multiple processing elements execute the same instruction across multiple data elements or threads. The multiple processing elements share a single program control flow unit and program counter and thus execute the same program but are able to execute that program with or using different data. In one example, each parallel processing unitincludes sixteen, thirty-two or sixty-four lanes, where each lane executes the same instruction at the same time as the other lanes in the parallel processing unitbut can execute that instruction with different data. Lanes can be switched off with predication if not all lanes need to execute a given instruction. Predication can also be used to execute programs with divergent control flow. More specifically, for programs with conditional branches or other instructions where control flow is based on calculations performed by an individual lane, predication of lanes corresponding to control flow paths not currently being executed, and serial execution of different control flow paths allows for arbitrary control flow.

132 138 138 The basic unit of execution in compute unitsis a work-item. Each work-item represents a single instantiation of a program or kernel that is to be executed in parallel according to the parallel processing paradigm employed. For example, in a SIMD architecture, multiple work-items execute the same instruction simultaneously on different data elements. Work-items can be executed simultaneously as a “wavefront” on a parallel processing unit, where each work-item executes the same instruction with different data and where different work-items can execute a different control flow path through the use of predication. In a SIMT architecture, work-items correspond to threads that can be executed simultaneously on the parallel processing unit, where different threads can execute different control flow paths. Threads are grouped into “warps” or “wavefronts”, which are scheduled or executed together.

138 138 138 102 138 138 138 136 132 138 For the purposes of this description, the term “wavefront” will be used, but it should be understood that this term broadly describes work-items that can be executed simultaneously and is inclusive of both “wavefronts” and “warps.” One or more wavefronts are included in a “work group,” which includes a collection of work-items designated to execute the same program. A work group can be executed by executing each of the wavefronts that make up the work group. In alternatives, the wavefronts are executed sequentially on a single parallel processing unitor partially or fully in parallel on different parallel processing unit. Wavefronts can be thought of as the largest collection of work-items that can be executed simultaneously on a single parallel processing unit. Thus, if commands received from the processorindicate that a particular program is to be parallelized to such a degree that the program cannot execute on a single parallel processing unitsimultaneously, then that program is broken up into wavefronts which are parallelized on two or more parallel processing unitsor serialized on the same parallel processing unit(or both parallelized and serialized as needed). A command processorperforms operations related to scheduling various wavefronts on different compute unitsand parallel processing units.

132 134 102 132 The parallelism afforded by the compute unitsis suitable for graphics related operations such as pixel value calculations, vertex transformations, and other graphics operations and non-graphics operations (sometimes known as “compute” operations). Thus in some instances, a graphics pipeline, which accepts graphics processing commands from the processor, provides computation tasks to the compute unitsfor execution in parallel.

132 134 134 126 102 116 The compute unitsare also used to perform computation tasks not related to graphics or not performed as part of the “normal” operation of a graphics pipeline(e.g., custom operations performed to supplement processing performed for operation of the graphics pipeline). An applicationor other software executing on the processortransmits programs that define such computation tasks to the APDfor execution.

100 103 105 116 202 204 103 102 105 102 116 202 132 116 204 132 204 103 204 202 204 105 100 100 In various examples, caches are present at one or more locations in the device. Some example locations include within the processor (e.g., processor cache), external to the processor (e.g., global cache), and, within the APD, as a global APD cacheor as a CU (“compute unit”)-local cache. In these examples, the processor cacheservices memory requests from the processor, the global cacheservices memory requests from a variety of units (e.g., processoror APD), the global APD cacheservices requests from the various compute unitsof the APD, and the CU-local cacheservices requests from the compute unitthat the CU-local cacheis within. In various examples, these different caches are arranged in a hierarchy, where caches at a lower level in the hierarchy (e.g., level-0 caches) are first consulted to satisfy memory requests, with caches that are higher in the hierarchy serving as a backing store to fill cache line misses (e.g., such higher-level caches are consulted in the event that a miss occurs in a lower-level cache). In an example, the processor cacheand CU-local cacheare lower-level caches, the global APD cacheis higher than the CU-local cache, and the global cacheis higher than any of these caches. In various examples, other caches not shown are present in the device. The present disclosure describes operations for a cache, and such operations can be performed for any cache of a computing device such as the device, including caches described herein or not explicitly described herein.

3 FIG. 300 300 302 304 310 302 102 138 304 308 310 308 306 302 308 310 310 illustrates a cache systemaccording to an example. The example cache systemincludes a processing unit, a cache, and a backing store. The processing unitis any processing unit (e.g., programmable processor or other circuitry, including the processoror a parallel processing unit) capable of making memory requests (e.g., asking to read or write from/to an address in memory). The cachehardware (e.g., circuitry) is capable of servicing such requests from an internal memory (cache storage), which has more limited storage area than the backing storebut has better access characteristics (e.g., less latency and/or better bandwidth). The cache storageis the memory that actually stores such cached data and the cache controlleris hardware (e.g., circuitry) configured to perform operations for managing the cache (e.g., updating cache line status, managing requests from the processing unit, sending requests to fill the cache storagewhen a miss occurs to the backing store, and so on). The backing storeis another (e.g., higher) level cache that can service requests for cache lines when a miss occurs or is system memory or another memory.

4 FIG. 308 304 302 304 304 308 304 304 illustrates the cache storageof the cache, according to an example. The present disclosure contemplates the use of “high priority” (also sometimes referred to as “high temporal”) cache lines. A cache line is the smallest unit of data that is tracked with metadata in the cache and that can be evicted or read into the cache. For example, in the event of a miss, meaning that data requested by the processing unitis not within the cache, the cachefetches the cache line containing that requested data and stores that cache line into the cache storage. Other information, such as replacement policy information, is stored on a per-cache line basis. In an example, a least-recently-used replacement policy is used in the cache. According to this policy, the cachetracks “age” information for various cache lines and determines which cache line to evict to make space for a new cache line that is to be stored into the cache (e.g., in the event of a miss), based on this age information. The age information is stored on a per-cache line basis, meaning that each item of age information indicates the age for one cache line.

304 306 308 “High priority” cache lines are cache lines that cannot be evicted. In other words, in the event that a miss occurs and the cachedetermines that a cache line is to be evicted in order to make room for the data for which a miss has occurred, the cache controlleridentifies a set of candidate cache lines for eviction. In some examples, these candidate cache lines are all of the cache lines in the set associated with the new cache line being stored into the cache (where “associated with” in this context means in the set specified by the address of the new cache line—the term “set” refers to a “set” in a set-associativity scheme, in which each cache line uniquely maps to a single set based on certain bits of the address of that cache line, and thus cache lines can only be evicted from that set). The set of candidate cache lines does not include, however, any cache line marked with “high priority,” meaning that no such cache line can be evicted. The present disclosure provides techniques whereby a portion of the cache storageis reserved for such high priority cache lines, and subsequent use of that portion to store cache lines marked as high priority.

304 304 In general, high priority cache lines cannot be evicted unless a high priority eviction condition is met. In some examples, this is true regardless of the priority of the incoming cache line being brought in as the result of a miss. In other words, as stated above, eviction occurs in response to a miss which needs a new cache line to be brought into the cache. In the present disclosure, the miss occurs as a result of a memory access for data not already in the cache, and this memory access can itself specify a priority. This priority would be the new priority for the cache line for the data being brought into the cache. Even if this priority is “high priority” for the new cache line, the cache line that is evicted cannot be a high priority cache line. Put differently, even incoming “high priority” cache lines cannot evict cache lines that are “high priority.” Thus, in general, cache lines having “high priority” cannot be evicted from the cache.

306 306 Above it is stated that a high priority cache line can be evicted in the event that a high priority eviction condition is met. Some examples of such a condition are now described. In one example, the cache controllermaintains an age for all cache lines including high priority cache lines. When a miss occurs in a set (a set-associative set), the cache controllerincrements a miss counter for that set. When the miss counter of a set arrives at a threshold, the oldest (or least-recently-used) high priority cache line in a set is converted into a regular priority cache line, which can then be evicted according to normal cache replacement policies. In another example of a high priority eviction condition, a timer based on clock cycles from the time the cache originally is brought into the cache, or a timer that measures real time from that point in time, is used. When this timer reaches a threshold, the oldest (or least-recently-used) high-priority cache line is turned into a regular priority cache line. In another example of a high priority eviction condition, a cache priority downgrade command, which may include cache writeback, flush, or invalidate commands, causes one or more of the high-priority cache lines to be turned into a regular priority line. For example, if a cache priority downgrade command targets - requests writeback, flush, or invalidation, of a cache line that is a high priority cache line, then that cache line is downgraded to a regular priority cache line (and the operations requested by the cache priority downgrade command is also performed). In another example, a power event, such as a request from a power or clock controller to put the cache into a lower-power state (such as a clock-gating or power-gating state) now or in the future causes one or more cache lines to be turned into regular priority cache lines. In another example, a cache coherence command causes a high-priority cache line to be turned into a regular-priority cache line. In another example, an error-checking calculation is periodically performed on cache lines. If the error-checking calculation shows that the value in a high-priority cache line has been corrupted, it is turned into a regular priority cache line (and in some examples is evicted). In another example, if the portion of the cache reserved for high-priority cache lines is changed, some or all of the high-priority cache lines in the cache are turned into regular priority cache lines.

308 304 306 304 304 306 306 A brief description of set associative caches is now provided. In a set associative cache, the cache storageis divided into sets. Any particular cache line is uniquely mapped to a single set based on bits of the address of that cache line. When a memory access request arrives at the cache, the cache controllerdetermines whether the cache line to satisfy that request is already in the cache. This is done by determining which set is mapped to the address of that cache line and then searching each cache line in that set for a match (e.g., by comparing a “tag”—a portion of the cache line address—of the cache line in the cache to a tag of the cache line being requested). If a match occurs, then a hit occurs and if no match occurs, then a miss occurs. In the event of a miss, the cacheselects a cache line in the set for eviction in order to make room for the new cache line). If there are no free slots (e.g., if there are no cache lines marked as “invalid”) in the set, then the cache controllerselects one of the cache lines in the set for eviction according to a replacement policy. In a common replacement policy - “least recently used” - the age of each cache line in a set is tracked. In this instance, “age” is not necessarily related to time per se, but to the order in which cache lines are brought into the set. For instance, the first cache line brought into a set is the “oldest,” the second is the next oldest, and so on. In a simple example, the cache controllerincrements an age of each cache line in a set when a miss occurs in that set, so that a higher age number means an older cache line. New cache lines are brought in with some initial value such as zero or another default value. It should be noted that the age is tracked among cache lines of a set, and relative ages of cache lines in different sets are generally not relevant (e.g., a cache line in one set cannot said to be “older” or “younger” than a cache line in a different set by comparing the ages of those two cache lines).

4 FIG. 308 402 404 302 304 304 302 304 304 304 402 304 404 illustrates a cache storagehaving an example reserved portionand a normal operation portionaccording to an example. Above, it is stated that the processing unitrequests the cacheto reserve a certain portion of the cachefor high priority lines. In some examples, this reservation is made by software executing on the processing unitsending a request for such reservation to the cache, where the request specifies an amount of space to be reserved. The cachethen stores an indication of which portion of the cacheis reserved for high priority cache lines (the reserved portion). The remaining portion of the cachecannot be used for high priority cache lines and is indicated as the normal operation portion.

302 304 304 308 304 304 402 308 402 402 402 In operation, when the processing unitsends a memory access request to the cache, the cachechecks whether the cache line for that request is already stored in the cache storage. If that cache line is not already in the cacheand the request specifies “high priority,” then the cachechecks whether there are available slots in the reserved portion(where a “slot” is a portion of the cache storagethat can store a cache line). A slot is available if there is at least one empty slot in the set mapped to by the cache line to be brought into the cache that is also in the reserved portion. In other words, the new cache line is mapped to a particular set. If there is at least one slot in that set that is both empty (e.g., marked as “invalid”) and within the reserved portion, then there is at least one slot available for the new cache line. If, however, there is no empty slot in that set that is also within the reserved portion, then the new cache line cannot be brought in as a high priority cache line, and the cache line is instead brought in as a regular priority cache line.

402 402 402 402 304 402 In some examples, the reserved portionis “soft reserved,” in that any amount of the reserved portionis permitted to be used for cache lines that are not high priority, but that such lines are evicted if needed to store high priority cache lines. In other words, while the reserved portionis not completely filled with high priority cache lines, the portions of the reserved portionnot used for high priority cache lines can be used for regular priority cache lines. However, once high priority cache lines are needed to be brought into the cache, the regular priority cache lines are evicted from the reserved portionto make space.

302 304 302 302 304 As stated above, a memory access request performed by the processing unitis transmitted to the cachefor processing. In some examples, these memory access requests are part of or embodied as instructions executed by the processing unit. More specifically, in various examples, the processing unitis a programmable processor that executes instructions. Some such instructions request access to memory (e.g., by loading values into a register, storing values from a register to memory, or otherwise reading from and/or writing to memory). Some instructions that request access to memory specify a priority hint. In various examples, this priority hint is included in the instruction itself, such as within the opcode (e.g., as a prefix), as one of the operands, or as any other component (set of bits) that is a part of an instruction. Such priority hints signal to the cachethat the cache line targeted by the instruction (e.g., the cache line at the address in memory written to and/or read from by the instruction) should have the specified priority.

304 306 304 304 402 402 304 304 304 304 404 In some examples, the priority specified by the instruction is a hint, meaning that the cacheis free to ignore that hint in certain circumstances. As described above, in some examples, the cache controllerobeys the specified hint unless an ignore condition is met. In one example, an ignore condition for an instruction specifying an address for a cache line and a priority is that the specified priority is “high priority,” there is a miss for the cache line in the cache, and there are no empty slots in the cachethat are within the reserved portion. In other words, if an instruction accesses a cache line not in the cache and provides a priority hint of “high priority,” but there is no room for that cache line in the reserved portion, then the cacheignores the hint to make the cache line high priority. In the event that the cacheignores the priority hint, the cachetakes an action other that setting the addressed cache line to the priority specified in the priority hint. In an example, if the priority hint for an instruction triggering a miss is “high priority” but there are no empty slots for the associated cache line in the reserved portion, then the cachesets the cache line to “regular priority,” so that the cache line can be stored into the normal operation portion.

304 304 306 402 306 402 In some examples, an instruction accesses an address for a cache line that is already in the cache. Further, the instruction specifies a priority hint for that cache line that is different than the priority already stored in the cachefor that cache line. In this situation, the cache controllermodifies the priority of that cache line to be the priority specified by the instruction. Stated differently, in the situation that a memory access results in an access to a cache line in the reserved portion, the cache controllersets the priority of that cache line to the priority specified in the instruction. Put differently, a cache line in the reserved portionhas the priority specified by the most recently executed instruction that specified a priority hint (unless that hint was not obeyed for some reason as specified elsewhere herein - for example, due to an ignore condition).

402 As can be seen, it is possible to modify the priority of a cache line that is already in the cache. Thus, a cache line can be “upgraded” (e.g., converted from “regular priority” to “high priority”) or “downgraded” (e.g., converted from “high priority” to “regular priority”) by executing an instruction that accesses data in that cache line and specifies the desired priority. For example, if a cache line is already in the cache and has a “regular priority,” then upgrading that cache line is performed by executing an instruction that accesses data in that cache line and specifies “high priority” (although an upgrade will not occur in some conditions, such as if the reserved portionhas no empty slots). Similarly, if a cache line is already in the cache and has a “high priority,” then downgrading that cache line is performed by executing an instruction that accesses data in that cache line and specifies “regular priority.”

304 It should be noted that the granularity with which priority data is stored is on a cache line basis. In other words, one item of priority data (i.e., one unit of information that specifies priority) is associated with one cache line. This means that accesses to different addresses in the same cache line can utilize and/or modify the same priority data. For example, a first access to a first address in a first cache line can cause that cache line to be stored into the cachewith one priority (e.g., high priority), and subsequent accesses to that cache line, even at a different address (but still within the cache line) act in accordance with that priority (e.g., a subsequent instruction that specifies an address in the same cache line but specifies a different priority will change the priority of that cache line).

5 FIG. 304 308 308 502 504 illustrates the cache, showing information within the cache storage, according to an example. As can be seen, the cache storagestores a number of cache lines, each with an associated item of priority metadata. The cache lines are processed in accordance with the teachings herein, respecting the associated priority metadata as described. In the example shown, each row has data for a cache line, as well as an associated item of priority metadata, which indicates the priority for the cache line within that row.

6 8 FIGS.- 6 FIG. 302 304 602 306 604 306 304 illustrate example interactions related to cache line priority.illustrates an interaction for reserving a portion of the cache for use for high priority cache lines. In this interaction, a processing unitexecutes an instruction that requests reservation of a portion of the cachefor use as a high priority portion (step). In response, the cache controllerreserves the requested portion (step). In various examples, the cache controllerperforms this reservation by storing an indication of which portion of the cacheis considered to be reserved for high priority.

7 FIG. 302 306 702 704 306 304 304 304 306 706 310 304 306 708 704 708 306 702 306 illustrates an interaction in which a memory access request is made that specifies a priority hint. The memory access request specifies an address that is for a cache line. The processing unittransmits such a request to the cache controllerfor processing. This is step. At step, the cache controllerchecks whether the access request results in a hit or a miss. This is done by checking whether the cache line for the specified address is within the cache. A hit occurs if the cachestores the cache line and a miss occurs if the cachedoes not store the cache line. If a miss occurs, then the cache controllerperforms step, loading a cache line from the backing storeinto the cache. The cache controllerperforms stepif a hit or a miss occurs at step. At step, the cache controllersets the priority for the cache line referenced by the memory access request (step). In particular, the cache controllersets this priority to the priority specified by the access request.

706 304 304 304 802 306 804 306 306 806 306 308 808 306 308 8 FIG. In some examples, step, which loads the cache line into the cache, causes an eviction. More specifically, if there are no available slots for the cache line being loaded into the cache, then an eviction occurs in order to store the cache line into the cache.illustrates an interaction for storing a cache line into the cachewhere an eviction occurs. At step, the cache controllerdetects an eviction event for a memory request that specifies a priority hint. An eviction event is an event that requires an eviction to occur. In one example, an eviction event occurs when a miss occurs for a memory access request and there are no empty slots for the cache line for that request. At step, the cache controllerselects a line for eviction. The cache controllerdoes not select a line that is considered “high priority,” but instead selects a cache line that has a different priority, such as “regular priority.” At step, the cache controllerevicts the selected line. In some examples, this eviction includes one or more of writing the cache line back to the cache storageif the cache line is dirty, invalidating the cache line, or performing other operations. At step, the cache controllerstores the new cache line from the backing store into the slot in the cache storagefrom which the cache line was evicted.

9 FIG. 3 FIG. 900 304 900 306 306 306 illustrates an example set associative cache, which is an example of the cacheof. The illustrated cacheincludes four sets, each having four ways, though an actual cache would have many more sets (and can have a different number of ways). In a set associative cache, any given cache line is mapped to a single set. When the cache line is to be brought into the cache, the cache controlleridentifies the set and then selects one of the ways in that set to place the cache line into. The combination of a set and a way is a “slot” in the cache. For example, set 1, way 1 stores one cache line, set 3, way 3 stores another cache line, and so on. When a cache line is to be brought into the cache (e.g., as the result of a miss), if there are no empty (invalid) ways in the set mapped to the cache line, then the cache controllerevicts one of the cache lines in one of the ways and places the new cache line into that slot. The manner in which such a cache line is selected for eviction is the replacement policy. In an example of a least recently used replacement policy, the cache controllermaintains an age for the cache lines in a set and selects a cache line for eviction based on this age.

306 In some examples, each set can include one or more cache lines that are marked as high priority and one or more cache lines that are marked as regular priority. In some such examples, the cache controllermaintains ages (or other replacement data) for each such cache line, but does not evict a cache line marked as high priority. In some examples, if a high priority cache line is downgraded to regular priority, the age of that cache line is available for use in the cache replacement policy.

10 FIG. 1 9 FIGS.- 1000 1000 is a flow diagram of a methodfor operating a cache, according to an example. Although described with respect to the system of, those of skill in the art will understand that any system configured to perform the steps of the methodin any technically feasible order falls within the scope of the present disclosure.

1002 306 306 306 306 402 10 FIG. At step, the cache controller sets a priority of a cache line to high priority based on a memory access request. More specifically, a memory request for an address is made, such as a memory request made as a result of an instruction requesting to access memory (e.g., a load or store instruction). This instruction specifies the priority for the cache line at the address. The cache controllersets the priority for that cache line. In some examples, the cache line is not in the cache (a miss occurs) and thus the cache controllerfetches that cache line, places that cache line into the cache, and sets the priority of the cache line to the priority requested by the memory access request. In other examples, the cache line is already in the cache and the priority for that cache line is different than the requested priority, and thus the cache controllersets the new priority for that cache line to the requested priority. Although the example ofis one in which the cache controlleris able to set the priority of the cache line to the requested priority, in some examples, this is not possible, such as where the reserved portionis full.

1004 306 1002 At step, the cache controllerdetects an eviction event. In some examples, the eviction event is that a subsequent memory access request is made for a cache line mapped to the same set as the cache line stored into the cache at step, and that set has no empty slots (e.g., no entries that are invalid). In this event, there is no room for this new cache line and thus one cache line from that set is evicted.

1006 306 306 306 At step, the cache controllerselects a cache line for eviction for the eviction event. This selection does not consider any cache lines for eviction that have “high priority.” In particular, an eviction event causes an eviction to occur in a particular set. In some examples, the cache controllerselects a cache line for eviction based on a replacement policy such as least recently used. In an example where the eviction event is that a new cache line is to be stored into the cache in a set where there are no empty slots, the set involved is the set mapped to that new cache line. The cache controllerselects one of the cache lines, but does not consider any cache line marked as high priority because such cache lines cannot be evicted.

306 306 Subsequent to this, the cache controllerevicts the selected cache line. In some examples, subsequently, the cache controllerstores the new cache line into the slot from which the older cache line was evicted.

308 402 404 402 404 402 404 402 402 308 306 308 Above it is described that the cache storageis divided into a reserved portionand normal operation portion. In some examples, this division is made by dividing one or more sets. In other words, for each set for which such a division is made, one or more of the ways of that set are included in the reserved portionand one or more other ways of the set are in the normal operation portion. In addition, in some examples, the size of the reserved portionis limited such that at least one way in each set is available for the normal operation portion, which thus allows evictions to be possible for all memory accesses. In other words, because high priority cache lines cannot be evicted, if any set had slots that were all within the reserved portion, and a high priority cache line was in each such slot, then no such cache line could be evicted for another cache line, and it would be possible for some cache lines to never be able to be stored in the cache (since cache lines uniquely map to sets). Thus in some examples, there is always at least one cache line in each set that is not within the reserved portion. This also means that the request to reserve a portion of the cache storagefor high priority cache lines is a hint and can be at least partially ignored (for example, if too much space is requested to be reserved, in some examples, the cache controllerlimits the amount of space in the cache storagethat is reserved).

304 304 Herein, the term “cache line” has one of several meanings based on context. When referring to data stored in the cache, a cache line is a unit of data stored in a slot in the cache. Cache lines in the cachehave an associated set of metadata, such as cache replacement policy metadata (e.g., age) and priority metadata. When referring to addresses, a cache line is a range of addresses (also called a “cache line address range”). An address can thus be said to be “within a cache line.” In general, cache lines are aligned to their size. This means that a certain number of the most significant bits of a cache line are all the same for all addresses within that cache line, with an offset beginning at 0 indicating an address within the cache line. The size of the offset is sufficient to specify any address within the cache line, and is based on the size of the cache line. For example, for cache lines that are 128 bytes, the offset is 7 bits, and the most significant bits (e.g., 57 bits in a 64-bit address scheme) specify the address of the cache line. An address is considered to be within the cache line if the address has identical most significant bits as the cache line.

It should be understood that many variations are possible based on the disclosure herein. Although features and elements are described above in particular combinations, each feature or element can be used alone without the other features and elements or in various combinations with or without other features and elements.

102 106 116 117 136 132 138 306 304 The various functional units illustrated in the figures and/or described herein (including, but not limited to, the processor, the auxiliary devices, the accelerated processing device, IO devices, the command processor, the graphics the compute units, the parallel processing units, or the cache controller(or other portions of the cache) may be implemented as a general purpose computer, a processor, or a processor core, or as a program, software, or firmware, stored in a non-transitory computer readable medium or in another medium, executable by a general purpose computer, a processor, or a processor core. The methods provided can be implemented in a general purpose computer, a processor, or a processor core. Suitable processors include, by way of example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), and/or a state machine. Such processors can be manufactured by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediary data including netlists (such instructions capable of being stored on a computer readable media). The results of such processing can be maskworks that are then used in a semiconductor manufacturing process to manufacture a processor which implements features of the disclosure.

The methods or flow charts provided herein can be implemented in a computer program, software, or firmware incorporated in a non-transitory computer-readable storage medium for execution by a general purpose computer or a processor. Examples of non-transitory computer-readable storage mediums include a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs).