Stochastic texture filtering

This application claims the benefit of U.S. Provisional Application No. 63/499,632 titled “Stochastic Texture Filtering,” filed May 2, 2023, the entire contents of which is incorporated herein by reference.

Two-dimensional (2D) image texture maps are essential to rich surface detail in most rendered images. Three-dimensional (3D) voxel grids play a similar role for volumetric effects like clouds, smoke, and fire, allowing detailed offline physical simulations to be used. The 2D image texture maps and 3D voxel arrays are widely used to add rich detail to the surfaces and volumes of rendered scenes, and filtered texture lookups are integral to producing high-quality imagery. Conventional high performance (in terms of quality and speed) texture filtering relies on precise computations that typically consume computation, power, and bandwidth resources. There is a need for addressing these issues and/or other issues associated with the prior art.

Embodiments of the present disclosure relate to stochastic texture filtering. Systems and methods are disclosed that enable efficient implementation of high-quality texture filters and efficient filtering of textures. Conventional texture mapping samples a texture map using an integer portion of the texture map coordinates to read multiple closest texels and applies a weight based on the non-integer portion of the texture map coordinates to each texel. The weighted texels are then combined according to a filter function to produce a color for each pixel.

In contrast to conventional texture map filtering, stochastic texture filtering introduces randomness into the texel sampling and/or filtering. Instead of computing a closest texel for the texture coordinates, randomness is introduced by stochastic sampling to obtain one texel. Stochastic sampling is also applied for filtering the texels when multiple samples are used and/or to perform temporal filtering. A first technique is used for discrete filters and filter-specific sample weights are generated. In contrast with conventional techniques, the sample weights are not applied directly to the single texel value. The single texel is randomly selected for each pixel, with probability proportional to an associated sample weight. A second technique is used for continuous filters and weights are not generated. Instead, the texture coordinates are perturbed with a random offset, which is drawn from a filter-specific probability distribution Stochastic texture filtering improves the performance of texture filtering in terms of speed and quality and is compatible with image reconstruction techniques such as DLSS (deep learning super sampling).

In an embodiment, the method includes receiving texture coordinates associated with rendered geometry that intersects a pixel, calculating integer texture coordinates for a texel stored in a texture map using the texture coordinates and a stochastic value that is based on a texture filter, and producing a filtered color for the pixel based on the texel.

Systems and methods are disclosed related to stochastic texture filtering. Stochastic texture filtering introduces randomness into the texel sampling and/or filtering. Instead of computing a closest texel for the texture coordinates, randomness is introduced by stochastic sampling to obtain one texel. Stochastic sampling is also applied for filtering the texels when multiple samples are used and/or to perform temporal filtering. A first technique is used for discrete filters and filter-specific sample weights are generated. In contrast with conventional techniques, the sample weights are not applied directly to the single texel value. The single texel is randomly selected for each pixel, with probability proportional to an associated sample weight. A second technique is used for continuous filters and weights are not generated. Instead, the texture coordinates are perturbed with a random offset, which is drawn from a filter-specific probability distribution Stochastic texture filtering improves the performance of texture filtering in terms of speed and quality and is compatible with image reconstruction techniques such as DLSS (deep learning super sampling).

illustrates a checkerboard texture rendered onto a plane using stochastic texture filtering for various filtering modes, including bilinear and bicubic filtering. A magnified regionand a minified regionfiltered using the various techniques are shown in a second and third row, respectively. For each pixel on a plane to which the checkerboard texture is applied, a single texel is sampled. A texel location, specified by (u,v) coordinates, is associated with geometry visible in the pixel and provided as an input. Various sources of randomness are available as an additional input for determining integer coordinates for the single texel. Magnification is handled using filter importance sampling.

On the left of, stochastic bilinear filtering and stochastic bicubic filtering is applied to one sample per pixel (1 spp), producing an incoherent texture for the minified region(middle row) and a grainy texture for the magnified region(bottom row). Use of DLSS improves the image quality, providing quality similar to hardware supported (HW) anisotropic filtering using stochastic bicubic filtering with 1024 spp that produces a high-quality reference image, shown in the rightmost images. Images reconstructed using DLSS are temporally stable, with occasional flickering in regions containing very high-frequency details. In motion, sporadic ghosting and other temporal artifacts introduced by DLSS are observed, but the overall image quality remains comparable to HW anisotropic filtering. Although DLSS doesn't completely remove noise caused by stochastic texture sampling, spatiotemporal blue noise reduces the noise, making it barely perceptible in magnified high-contrast areas.

In the center of, stochastic bilinear filtering and stochastic bicubic filtering is applied at one sample per pixel (1 spp) and DLSS is used, producing a coherent texture for the minified region(middle row) and a antialiased texture for the magnified region(bottom row). When the stochastic bilinear and bicubic filtering is used with DLSS, the results are very similar to the high-quality reference images. In sum, temporal reconstruction with stochastic texture filtering is effective in recovering a high-quality anisotropically filtered image while only using 1 spp.

More illustrative information will now be set forth regarding various optional architectures and features with which the foregoing framework may be implemented, per the desires of the user. It should be strongly noted that the following information is set forth for illustrative purposes and should not be construed as limiting in any manner. Any of the following features may be optionally incorporated with or without the exclusion of other features described.

Textures are given as discrete, uniformly-spaced samples. Filtering texture lookups is challenging since each access generally requires a spatially-varying anisotropic filter that accesses multiple source texture samples. Two main types of texture filtering may be used: interpolation for translation and magnification, and lowpass filtering for minification. For both texture filtering types, the notation of a filter function ƒ is defined over the texture-space coordinates domain R→R and (u, v, . . . ) as its inputs. Without loss of generality, a simplified, two-dimensional notation is used due to the separability of the sampling process. The filtered texture value is an integral of the product off and the texture lookup function t:

The texture function t is defined everywhere, but is non-zero only at discrete locations (typically uniform grid) due to impulse train sampling.

Two different techniques may be used to perform stochastic texture filtering based on the integral of Equation (1). A first technique is used for discrete filters and sample weights are generated. However, in contrast with conventional techniques, the sample weights are not applied directly to the single texel (texture value). Instead, in an embodiment, a list of texel coordinates and associated sample weights is generated for a given discrete filter. A single texel is randomly selected for each pixel, with probability proportional to a texel weight,

illustrates sixteen texels and associated sample weights, in accordance with an embodiment. When 1 spp is used, one of the sixteen texels will be sampled. In an embodiment, the list of texel coordinates and weights are not all stored in memory. In an embodiment, reservoir sampling is used to compute individual texel weights. For each texel, a decision is randomly made whether to discard the sampled texel or to replace the current texel with the sampled texel.

illustrates sixteen texels and associated sample weight probabilities, in accordance with an embodiment. The sample weights shown inare converted into the probabilities shown in. After all texels have been processed for multiple samples and/or frames, the texels will each be selected with probability proportional to their associated weight. While the sample weight is not directly applied to the selected texel, a larger sample weight will cause the associated texel to be selected more frequently, and when averaged over many iterations, the resulting texel will converge to the correct estimated value.

illustrates stochastically selected texels and associated sample weight probabilities, in accordance with an embodiment. For first and second frames (or different samples for a pixel in the same image or frame), sample texelsandassociated with 20% and 10% probabilities, respectively, are selected. Over several frames, the texels associated with 20% probability will be stochastically selected more frequently compared with the texels associated with 10% or 0% probability. In contrast with conventional texture filtering, there is no explicit weighting of texels or combination of texel values according to a filter function. In an embodiment, texel values are averaged when multiple samples are used and/or to perform temporal filtering. The texture filtering being used (in the limit) is determined by the sample weight values associated with the texels.

The sample weights for discrete texture filtering may be generated for separable functions, weighted sums, uniform sample reuse, sampling arrays, weighted reservoir sampling, positivization, and the like. In the following, denotes uniform random variables in [0,1) and angled brackets to denote expectation. For separable functions, an n-dimensional function that is a product of one-dimensional (1D) functions can be sampled by independently sampling each dimension. Many filters used for textures, including Gaussian and polynomials (linear, cubic, etc.) are separable. For weighted sums the simplest texture filtering function ƒ can be represented as a set of discrete weights w defined for multiple discrete texture samples t. Given normalized weights wand texture values t, the filtered texture value is given by

If a term j of the sum is sampled with probability equal to w, then an unbiased estimate of F is given by the corresponding texture value, unweighted:

Under the assumption that ware normalized, this is a special case of sampling a term according to probabilities p∝w, and applying the standard Monte Carlo estimator

The weights w, are often not normalized, and so must be normalized to find weights

before filtering. However, in this case, normalization may be skipped, and j may be sampled with probability proportional to wand Equation (3) may be applied to get the correct result.

In an embodiment, a single 1D random number is used for sampling each dimension and is remapped to a fresh uniform sample for each dimension to perform uniform sample reuse. Whenever a 1D random variable ξ is used to make a discrete sampling decision based on a probability p, then a new independent random variable ξ′∈[0,1) can be derived from ξ:

This technique can be useful when is well-distributed (e.g., with a blue noise spectrum or with low discrepancy), allowing additional dimensions to benefit from ξ's distribution as well as saving the cost of generating additional random samples.

For a sampling array, an array of weights w, (as from Equation 2) can be sampled by summing the weights and selecting the first item j where

For weighted reservoir sampling storing or recomputing all of the weights wmay be undesirable, especially on GPUs. Weighted reservoir sampling with sample reuse can be applied with weights generated sequentially.

Although negative weights can be sampled with probability based on their absolute value, doing so does not reduce variance as well as importance sampling does with positive functions. All interpolating filters of a higher order than the linear filter have negative lobes and being able to estimate them with low variance is essential for stochastic texture filtering. Positivization may be used, partitioning the filter weights w, into positive

and negative

sets and sampling once from each set. Given respective sample indices jand j, the estimator of the filtered texture value of Equation (2) is

If the original filter was normalized, the resulting positive and negative parts will not be normalized, and if Equation (2) is used, both sums need to be weighted.

A second technique is used for continuous filters (without discretization) and weights are not generated. Instead the (u, v) coordinates are perturbed with a random (u′, v′) offset, which is drawn from a probability distribution. The choice of distribution used to draw random offsets ultimately determines the effective texture filtering (once again, in the limit, after accumulating many samples).

illustrates continuous distribution sampling, in accordance with an embodiment. A probability distributionis used to determine the random offset that is applied to the texture coordinates. Texture coordinates are offset using a random offsetthat is drawn from the probability distribution.

illustrates texel sampling using random u,v offsets, in accordance with an embodiment. A sample texelis selected for offset texture coordinates corresponding to the random offset. A sample texelis selected for offset texture coordinates corresponding to another random offset drawn from the probability distributionand no weights are used.

For a filtering operation given by the product of a normalized continuous convolutional filter ƒ(u, v) with a texture t(u, v) expressed in the form of Equation (1), an unbiased estimate of F can be found using filter importance sampling (FIS): (u′,v′) is sampled from ƒ(u, v)'s distribution and the standard Monte Carlo estimator is applied, giving (F)=t(u′,v′). This approach is appealing for stochastic texture filtering since it allows for filters with infinite spatial support and doesn't have a cost that necessarily scales with the filter's width. The FIS framework can be used with positivization for low variance evaluation of filters with negative lobes.

Filter importance sampling a screen-space reconstruction filter can effectively approximate a minification filter, such as an anisotropic filter. However, FIS is not sufficient to perform UV jittering for magnification, as it would produce nearest-neighbor interpolated texture and visual artifacts. FIS may be used for texture reconstruction and sampling in addition to screen-space reconstruction filtering.

FIS assumes the integration of a product of two continuous functions. When using FIS to filter discrete samples, a practical realization draws a sample x′ from ƒ and then selects the closest texel [x′+½]. For n-dimensional filtering, this corresponds to applying a box reconstruction filter over [−½,½]to the texture to make a continuous function t(x). Equivalently, applying the box reconstruction filter corresponds to convolving the original filter function ƒ with a box filter, changing the shape of the filter function. Thus, the filter function that is sampled should be the deconvolution of the desired filter with the box function. This perspective provides a better understanding of Hofmann et al.'s stochastic trilinear sampling algorithm, which is based on independent, uniform jittering in each dimension and then applying nearest neighbor sampling. The jittering corresponds to applying FIS to sample the box filter which is then convolved with another box function, giving a stochastic trilinear interpolant.

Thus, a B-spline filter of degree n may be filtered by sampling a spline of degree n−1 and performing a nearest lookup, since approximating B-splines are constructed by repeated convolution of a box filter via the Cox-de Boor recursion formula. For example, a quadratic B-spline filter can be achieved by sampling a triangular probability density function (PDF) over [−1.5, 1.5]. Sampling can either be performed via cumulative distribution function (CDF) inversion or by adding n uniformly-distributed random variables.

The additional box function can be useful for rapidly changing filters such as a small-sigma Gaussian: evaluating it at discrete points results in subsampling error and the correction requires evaluating the erf error function. Filter importance sampling a regular, analytical normal distribution produces the same effect due to the convolution of nearest-neighbor box function with the Gaussian. Furthermore, a Gaussian convolutional filter is an example of an infinite filter that is truncated in practice. With FIS, it is possible to evaluate an infinite filter by sampling the filter without truncation. This can simplify implementation (it is not necessary to carefully window the filter), as well as saving the computational cost of multiple discrete weight evaluations and sample selection.

Hardware support for stochastic texture filtering, using discrete and/or continuous filter functions, may be implemented within a texture map unit.illustrates a block diagram of an example stochastic texture unitsuitable for use in implementing some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities may be carried out by hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. Furthermore, persons of ordinary skill in the art will understand that any system that performs the operations of the stochastic texture unitis within the scope and spirit of embodiments of the present disclosure.

The stochastic texture unitincludes a stochastic filter sample unit, texture combining unit, and interfaces to a texel storage. In an embodiment, the texel storageis a cache hierarchy backed by a local on or off device memory storage. The stochastic texture unitreceives texture coordinates for a pixel. When multiple texels are sampled for each pixel, texture coordinates may be received for each sample. The stochastic texture unitcomputes a filtered texel (color) for each pixel. A sampling mode (number of samples per pixel) and filtering mode (discrete or continuous) may be specified for each texture map or provided via an instruction (per texture/object/image/etc.).

The sampling mode and filtering mode may contribute to computations of stochastic random values, sample weights, and texel coordinates (for one or more texels per pixel). The filtering mode specifies the discrete or continuous filter modes and the sample weights or probability distribution, respectively. The stochastic filter sample unitcomputes the integer texel coordinates for each texel according to the sampling and filtering modes. The sampling mode may define sampler state for controlling the number of texels selected and the coordinate computation, where the sampler state may comprise spp, maximum degree of anisotropy in mip level selection (e.g., from 16 to 64 or higher for improved minification filter quality), and selecting from a number of built in filters, including but not limited to polynomial (e.g., cubic B-spline or Mitchell filter) or the Gaussian filter. The sampling mode may define sampler state for using an average value of the entire texture to stochastically skip texel fetches entirely. The sampling mode may define sampler state for providing a texture priority number for adjusting the degree of stochasticity to maintain performance, so that when performance drops textures with lower priority are more aggressively sampled stochastically.

In an embodiment, the filtering mode may define a filter kernel for stochastic sampling using a number of polynomial coefficients (e.g., cubic B-spline polynomial, Mitchell polynomial, and the like). In an embodiment, the filtering mode may define a filter kernel for stochastic sampling using a parametric function (e.g., Gaussian, windowed sine, B-spline, program, etc.) with a number of user-specified coefficients that control the function's shape. In an embodiment, the filtering mode may define a filter kernel for stochastic sampling by a user-provided lookup table. In an embodiment, the filtering mode may define a filter kernel for stochastic sampling via a program that is executed to compute texture filtering coefficients at a specified lookup point. In an embodiment, the same filter kernel is used for all texture dimensions. In an embodiment, independent filter kernels are specified for each texture dimension.

The filtering mode may also specify a sampling filter, such as using weighted reservoir sampling to sample from a set of filter weights or filter importance sampling to sample an analytic filter function. When FIS is used with a Gaussian filter, better results may be produced with narrow filters that would otherwise “fall between the samples.” The filtering mode may also specify the use of positivization to sample filters with negative coefficients, randomly selecting between the positive and negative samples, reweighting the samples such that the result remains correct in expectation. The filtering mode may also specify stochastic sampling of a single texture level of detail (LOD) for 1-3D textures, such as stochastic quadrilinear sampling for MIP mapped 3D textures or stochastic sampling may be specified via a “jitter” value provided by the user that is added to the hardware LOD before the LOD is rounded to an integer MIP level. The filtering mode may also enable/disable clamping of highly anisotropic filters.

The stochastic filter sample unitobtains a random number for stochastic texture sampling (reading an approximate texel instead of computing the “correct” texel) and/or stochastic texture filtering (via the sample weights). The random number may be provided by a user, hardware-generated, or read from a table (fixed or programmed). The random number may be a low-discrepancy random number or selected from a particular distribution (e.g., spatiotemporal blue noise). A single 1D random sample may be used for sampling all dimensions, remapping the 1D random sample to a fresh uniform sample in [0,1) after each dimension is sampled.

In an embodiment, the integer texel coordinates associated with a sample weight are selected, where the sample weight is stochastically selected with a probability proportional to the sample weights for the discrete filter. In an embodiment, a random offset (u′, v′) is drawn from a distribution and summed with the fractional (u, v) to produce integer texture map coordinates.

The texel combining unitreceives texels from the texel storageand the sampling mode. When multiple samples are used for a pixel, the texels are combined to produce a filtered texel for each pixel. When a single texel is read for each pixel, the texel combining unitsimply outputs the texel read from the texel storageas the filtered texel.

Equation (8) describes how randomness may be used for a linear filter. More specifically, direct application of sampling using an array of weights and then Equation (3) gives the following estimator for linear interpolation over [0,1], lerp(v, v, t)=(1−t) v+tv:

Bilinear interpolation of values at the four corners of the unit square, bilerp(v, v, v, v, s, t), can be implemented with nested linear interpolations. Applying the same approach and reusing the sample, results in: