YUV 4:4:4 encoding using chroma subsampling

While captured video imagery and computer-rendered video imagery typically is generated using a red-blue-green (RGB) or similar color space and is also displayed using an RGB color space, the encoding of such video imagery often utilizes a color space conversion to a luminance-chrominance, or YUV, color space in order to take advantage of certain characteristics of the human vision system (HVS). A YUV color space specifies, for each pixel or other picture element, a luminance component (Y) and two differential chrominance components: blue projection (U) and red projection (V) (and from the combination of which the green chrominance can be calculated). The array of Y components for an image (the “Y plane”) thus is a monochrome representation of the image, and the arrays of U and V components for the image (collectively, the “UV plane”) thus represent the color information for the image.

In the HVS, color information (chrominance) is processed at a much lower resolution than luminance. Accordingly, many video encoding/decoding systems leverage this disparity for improved encoding efficiency by subsampling the chrominance information while still maintaining high decoded image quality. For example, in a YUV 4:2:2 encoding scheme, the full luminance resolution is maintained (as indicated by the first “4” in “4:2:2”) while only one-half of the chrominance resolution is maintained (as indicated by the “2:2” in “4:2:2”), typically by maintaining all rows of the UV plane but only one-half of the columns of the UV plane, and thus reducing the total amount of image data to be encoded by one-fourth. Still further, in a YUV 4:2:0 encoding scheme, the full luminance resolution is maintained while only one-fourth of the chrominance resolution is maintained (as indicated by the “2:0” in “4:2:0”), typically by maintaining only one-half of the rows and one-half of the columns of the UV plane, and thus reducing the total amount of image data to be encoded by three-eighths.

YUV 4:2:2 and YUV 4:2:0 subsampling often yield an acceptable user experience for display of video with considerable motion (as the HVS does not have enough time to process the details) or with video content with slowly-changing colors and relatively few sharp edges. However, the negative impact of such chrominance subsampling on certain kinds of video content, such as the display of text (and particularly when on a colored background) or the display of imagery with many fine lines (such as certain types of art or technical drawings), becomes perceptible to users and thus results in a degraded user experience.

YUV 4:2:2 and 4:2:0 encoding schemes (referred to herein collectively as “sub-4:4:4 encoding”) provide advantageous reductions in the amount of image data to be encoded, transmitted, and decoded, at the expense of reduced chrominance resolution. In many instances, this reduced chrominance resolution has little impact on the perceived display quality of the video content due to the human vision system's reduced sensitivity to chrominance (or “chroma” for short) compared to luminance. However, certain image content, such as the aforementioned text content or fine line content, suffers from chroma subsampling. In such instances, full YUV 4:4:4 encoding would be employed. However, some video systems only have native support for sub-4:4:4 encoding. For example, a source device that operates to encode video content and transmit the resulting encoded video content may employ a hardware encoder that only natively supports 4:2:2 or 4:2:0 encoding; that is, is configured to always apply chroma subsampling or to accept only chroma-subsampled inputs. As another example, a sink device that operates to receive encoded video content and then decode the encoded video content to recover a representation of the original video content for display or for further processing may employ a software-based decoder that only natively supports 4:2:2 or 4:2:0 decoding. In such systems, YUV 4:4:4 video cannot be successfully encoded and decoded to recover YUV 4:4:4 video content.

The following describes implementations of systems and methods for providing YUV 4:4:4 support in a video system in which one or both of the encoder or decoder have only native support for sub-4:4:4 encoding/decoding through a process referred to herein as subframe-based chroma subsampling. In implementations, the subframe-based chroma subsampling scheme operates to effectively induce YUV 4:4:4 encoding/decoding end-to-end even when one or both of the encoder circuit or decoder circuit are limited to sub-4:4:4 encoding by generating, at a source device, multiple subframes from the YUV data for an input image to be encoded. Each subframe contains the full luminance resolution (that is, a copy of the entire Y plane) of the image, and a different subset of the chroma resolution (for example, a separate corresponding portion of the UV plane), wherein each different subset of the chroma resolution differs from the other subsets of the chroma resolution by at least one chroma value of the chroma resolution. Each subframe then is processed independently by the encoder as though it were a separate chroma subsampled image, with the result being a different encoded video bitstream (or portion thereof) that contains an encoded representation of the entire Y plane and a corresponding separate portion of the UV plane of the original image. As only a subsample of the full chrominance resolution is present in each subframe, each subframe can be properly encoded by an encoder that natively supports only sub-4:4:4 encoding.

The encoded bitstream(s) are transmitted to a sink device, whereby a decoder circuit decodes the encoded bitstream(s) to recover multiple copies of the full Y plane and a copy of each of the respective portions of the UV plane included in each encoded subframe represented in the corresponding bitstream. As each subframe has less than full chrominance resolution, the corresponding encoded bitstream can be decoded by a decoder that only natively supports sub-4:4:4 decoding to generate a decoded subframe. A post-decoding process then operates to regenerate a representation of the original image by utilizing the decoded full Y plane from one of the decoded subframes and by compositing the full UV plane from the individual respective UV plane portions obtained from the decoded subframes. Thus, a YUV 4:4:4 image can effectively be encoded and decoded in a system having one or more components in the pipeline that are limited to native support of only sub-4:4:4 encoding/decoding through generation of multiple subframes from the original YUV plane that each have less than full chroma resolution and thus can be properly handled by every stage in the pipeline.

In the following, reference is made to “circuit,” which should be understood to reference any or a combination of a hardcoded circuit (e.g., a corresponding portion of an application specific integrated circuit (ASIC), a programmable circuit (e.g., a corresponding portion of a field programmable gate array (FPGA) or programmable logic device (PLD), or one or more processors executing software instructions that cause the one or more processors to implement the ascribed actions. Reference also is made to “YUV”, which shall be understood to refer to any of a variety of luminance-chrominance color spaces (also known as “color models”), including those using linear luminance or non-linear luminance (Y′), such as YCbCr, Y′CbCr, YPbPr, Y′PbPr, Y′IQ, YDbDr, and the like.

illustrates a video systememploying chroma subsampling to provide YUV 4:4:4 support using native sub-4:4:4 encoding/decoding in accordance with some implementations. In the illustrated example, the video systemincludes a source deviceconnected to a sink devicevia a transmission medium, which may represent, for example, a wired cable, a wireless connection, a complex network (e.g., a wireless local area network (WLAN) and/or the Internet), and the like. An example of a hardware implementation of one or both of the source deviceor the sink deviceis described in detail below with reference to.

The source deviceoperates to receive a stream of video images (e.g., input image) from, for example, a camera (captured imagery) or as a result of an video rendering process at the source deviceor another device upstream, encode each video image to generate an encoded representation of the video image, and transmit the encoded representation to the sink device. The sink devicein turn operates to receive the encoded representation of each video image via the transmission mediumand then decode the encoded representation to obtain a corresponding output video image (e.g., output image), which represents the content of the corresponding input video image. The output video image then may be provided for display, stored for subsequent access, further processed (e.g., encoded at a different resolution), and the like.

To perform the encoding process, the source deviceincludes a color space conversion circuitto convert the input imagefrom an RGB color space to a YUV color space (in the event that the input imageis not already in a YUV color space compatible with the encoding process) and an encoder circuitto encode the image using any of a variety or combination of encoding schemes, such as H.264, H.265, AV1, Motion JPEG, and the like. Likewise, to perform the decoding process, the sink deviceincludes a decoder circuitto decode the encoded data representative of the input imageto generate a YUV image and, if conversion to an RGB color space is needed, a color space conversion circuitto convert the YUV image to an RGB color space, resulting in the output image. However, in implementations, one or both of the encoder circuitor the decoder circuitis unable to natively support full chroma resolution processing (that is, YUV 4:4:4). For example, the encoder circuitmay be fully implemented in hardcoded logic that is not configured to support YUV 4:4:4 and thus utilizes a hardware-based or software-based native chroma subsampling processthat automatically chroma subsamples any input YUV image to sub-4:4:4 resolution (e.g., to 4:2:2 or 4:2:0 resolution). As another example, the decoder circuitmay be limited by its hardware or software configuration to only be able to process image data that has been encoded at a sub-4:4:4 resolution. Thus, unmodified operation of the video systemwould result in the transmission of encoded image data that has been chroma subsampled (at YUV 4:2:2 or 4:2:0) and, by extension, decoding of image data that has no more than 4:2:2 or 4:2:0 resolution.

To avoid mandatory chrominance resolution loss and to provide full YUV 4:4:4 encoding support in the video system, in implementations the source deviceand sink deviceemploy a subframe-based chroma subsampling scheme that, in effect, distributes the full chrominance resolution (that is, the full UV plane) of the input imageover multiple subframes, each of which is treated as a separate image by the encoder circuitand the decoder circuit. At the source device, a chroma preprocessing circuitoperates as an interface between the color space conversion circuitand the encoder circuit, while at the sink device, a chroma postprocessing circuitoperates as an interface between the decoder circuitand the color space conversion circuit. As explained in more detail below with reference to, the chroma preprocessing circuitand postprocessing circuittogether implement the subframe-based chroma subsampling scheme so as to generate a plurality of subframes such that even after the chroma subsampling processis applied to each subframe by the encoder circuit, the resulting plurality of encoded subframes (as respective bitstreams) together, or collectively, contain the full chroma resolution of the input image(or in some implementations, a higher chroma resolution than otherwise is natively supported by the encoder circuitor decoder circuit), and from which a decoded representation of the input imageis composited to obtain the output image. As such, the circuitsandtogether can operate to provide YUV 4:4:4 support in the video systemthat has at least one of the encoder circuitor the decoder circuitwith native sub-4:4:4 support only.

illustrates a methoddepicting an example encoding operation of the source devicein more detail in accordance with implementations. To facilitate understanding, the methodis described with reference to an example of an encoding process depicted by. The methodis initiated by generation or receipt of an RGB image (input image) at the source device. For example, the source devicemay be connected to a video camera or other imaging system that is capturing live video as a sequence of RGB images. As another example, the source devicemay include or be connected to a rendering system that is rendering video as a sequence of computer-generated images (e.g., as a remote video game server). As yet another example, the source devicemay be accessing pre-recorded or pre-rendered video content from local or remote storage (e.g., as a video streaming server).

In response to receipt or access of the input image, and if the input imageis not already in a compatible YUV color space, then at blockthe color space conversion circuitoperates to perform color space conversion by converting the RGB color space of the input imageto a YUV color space using any of a variety of well-known or proprietary color space conversion schemes, resulting in a full chroma resolution YUV image(). Otherwise, if already received in a compatible YUV color space, the process of blockis bypassed and the input imageand the YUV imageare the same image. The YUV imagecomprises pixel data composed of a Y plane() comprising the luminance values for the pixels of the input imageand a UV plane() comprising the U chrominance and V chrominance values representing the pixels of the input image. It will be appreciated that the UV planeitself can be represented as a U plane and a V plane, but is illustrated and referenced in aggregate as the UV planefor ease of reference and consistent with industry norms. In implementation, the input imagetypically would have many thousands or millions of pixels and thus the resulting Y plane and UV plane would have many thousands or millions of values, but for ease of illustration the example Y planeand UV planeofare depicted and described using a simple example of 8×8 arrays for each, representing an 8×8 pixel array for the input image.

At block, the chroma preprocessing circuitoperates to generate up to four subframes-(also denoted as subframes-) () from the YUV image, each subframe having the same entire Y planeand a distinct portion of the UV plane, which is referred to herein as a “UV plane portion”. In some implementations, the distinct UV plane portions included in subframes-are generated via selective chroma sampling of the UV planethat is offset in position for each subframe relative to the sampling performed for the other subframes. For example, as illustrated in, the chroma preprocessing circuitdetermines a UV plane portion for inclusion in the subframeby using zero pixel offset in the starting position for the corresponding sampling. The chroma preprocessing circuitdetermines a UV plane portion for inclusion in the subframeby applying a downward pixel offset in the starting position for the corresponding sampling, such that the UV plane portion includes all of the values of the UV planeexcept the top row. The chroma preprocessing circuitdetermines a UV plane portion for inclusion in the subframeby applying a leftward pixel offset in the starting position for the corresponding sampling. The chroma preprocessing circuitdetermines a UV plane portion for inclusion in the subframeby applying a downward-and-leftward pixel offset in the starting position for the corresponding sampling. Other orders of offset sampling instead may be employed, as well as other directions of offset sampling (e.g., offsetting the starting position for sampling up instead of down and/or offsetting right instead of left).

At blocks-to-, the resulting subframes-, respectively, are supplied to the encoder circuit, which treats the copy of the Y planeand distinct UV plane portion in each subframe as a separate image for encoding. To illustrate with reference to block-, the corresponding copy of the Y planeand the UV plane portion are input to the encoder circuit, which at sub-blockemploys the sub-4:4:4 chroma subsampling process to chroma subsample the UV plane portion (e.g., at a 4:2:2 subsampling) to generate a corresponding subsampled UV plane portion, while at sub-blockthe corresponding copy of the Y planeis maintained without subsampling. This process is repeated for each of the remaining subframes-to-at the corresponding blocks-to-, either in parallel or in sequence, depending on the configuration and resources of the encoder circuit.

In implementations, the chroma subsampling processoperates to subsample each input UV plane portion using the same subsampling process or algorithm. For example, if employing a 4:2:0 subsampling, the chroma subsampling processmay obtain a 50% reduction in both horizontal and vertical resolution by discarding the UV value at every other row and every other column, such that the first UV value at the first row and first column in the UV plane portion (that is, the first U value from the U plane and the first V value from the V plane) is maintained, but the second UV value at the first row and second column is discarded, and so forth. That is, the subsampling performed by the chroma subsampling processemploys the same maintain/discard pattern for each input UV plane portion, starting at the same start/reference point of the first UV value at the first row and first column. Thus, by performing the four different chrominance subsampling on the original UV planeto generate the four distinct UV plane portions, the maintain/discard subsampling pattern is applied at a different starting point relative to the original UV planesuch that the full chroma content of the original UV planesubsequently is present in aggregate across all four subsampled UV plane portions.

To illustrate, refer to a 2×2 arrayof luminance values (denoted L, L, L, and L) from the Y planeand a corresponding 2×2 arrayof chrominance values (denoted C, C, C, and C) from the original UV plane. As a full copy of the original Y planeis maintained by the chroma subsampling processfor the subframes-, each of the resulting subsampled subframes-(also referred to herein as subsampled subframes-, respectively) includes a full copyof the 2×2 arrayof luminance values (or every second or fourth subsampled subframe, so as to reduce the amount of time needed to encode and decode the full image consisting of a number of chroma-subsampled subframes. However, in this example, a 4:2:0 subsampling process is employed, and thus for every separate 2×2 array of UV values in an input UV plane portion, only the top-left-most UV value is maintained, and the other three UV values are discarded (assuming, in this example, the first row and first column are maintained in the maintain/discard subsampling pattern). Thus, through the offset sampling process, the UV values in each 2×2 array are different for each of the four UV plane portions generated from the offsetting of the starting point for sampling of the UV plane. Thus, with reference to the 2×2 array, for subframe, the chroma value Cis maintained while chroma values C, C, and Care discarded. For subsampled subframe, the chroma value Cis maintained while chroma values C, C, and Care discarded. For subsampled subframe, the chroma value Cis maintained while chroma values C, C, and Care discarded. For subsampled subframe, the chroma value Cis maintained while chroma values C, C, and Care discarded. As a result, while the resulting subsampled UV plane portion for each subsampled subframe-does not individually have the original resolution of UV plane(and thus is capable of being encoded by the encoder circuitwith only native sub-4:4:4 support), the subsampled subframes-collectively contain all of the original chroma information of the UV planeeven after the lossy chroma subsampling processis applied to each subframe. For example, with reference to the 2×2 arrayfrom the UV plane, the subsampled subframes-together contain the chroma values C, C, C, and C, and thus from subsampled subframes-the complete 2×2 arraycan be recreated even though each individual subsampled subframe contains only one chroma value for the 2×2 array. This same result occurs for each 2×2 array in each of the Y planeand the UV plane.

Referring back to, following application of the chroma subsampling processto each of the subframes-to generate the subsampled subframes-, the subsampled subframes-are encoded by an encoding process() at the encoder circuit, as represented by blocks-to-, respectively, to generate corresponding bitstreams-(), respectively, with each bitstream representing an encoded representation of a corresponding subsampled subframe. Thus, for a given input subsampled subframe, the encoding processincludes encoding the full copy of the Y planeand encoding the subsampled UV plane portion of the subsampled subframe to generate a bitstream. Any of a variety of encoding schemes may be employed, such as H.264, H.265, AOMedia Video 1 (AV1), and the like. At block, the resulting plurality of bitstreams-are transmitted to the sink devicevia the transmission medium.

In some implementations, such as for H.264, H.265, or AV1-based codecs, each of the subframes-is encoded as a separate encoder session (substream) and decoded as a separate decoder session (substream) so as to avoid flicker or other artifacts. The multiple sessions may be implemented using multiple instances of codec hardware in parallel, or time-multiplexing the same codec hardware between the multiple sessions. In other implementations, such as for Motion JPEG-based codecs, which encode each image of the video stream independently of other images and do not cross-reference images, all of the subframes may be encoded in the same video stream and the encoder circuitcould provide a descriptor or other metadata that identifies which chroma data is associated with which image. Moreover, while the description above describes a process in which the luminance information is included in each subframe for purposes of compatibility with common types of codecs, in an alternative implementation, the encoder circuitcan be configured to incorporate the luminance information in only every Xth subframe (X=2 or 4) and the decoder circuitbeing configured to expect and extract the luminance information from only every Xth subframe so as to reduce the amount of encoded duplicate luminance information transmitted between the source deviceand the sink device.

illustrates a methoddepicting an example decoding operation of the sink devicein more detail in accordance with implementations. To facilitate understanding, the methodis described with reference to an example of a decoding process depicted byand which represents the decoding of the encoded subsampled subframes-illustrated in the example of. The methodis initiated at blockwith the receipt of the bitstreams-(bitstreams-) from the source device. At blocks-to-, the decoder circuitdecodes each of the bitstreams-to generate corresponding decoded subframes-(decoded subframes-,). The decoded subframes-represent either exact copies or inexact copies of the subsampled subframes-(subsampled subframes-,) depending on whether the encoding/decoding processes employed by the encoder circuitand decoder circuitare lossless or lossy.

Following decoding, the decoded subframes-individually have sub-4:4:4 resolution (and thus are able to be decoded by an implementation of the decoder circuitthat does not have native support for YUV 4:4:4 encoding). However, the decoded subframes-collectively contain all of the original chroma information of the UV plane. Accordingly, at blockthe chroma postprocessing circuitoperates to generate a representation of the original input imagein the YUV color space by compositing the full UV planefrom the subsampled UV plane portions extracted from the plurality of decoded subframes-. Moreover, the entire Y plane may be extracted from one or more of the decoded subframes-. For example, in some implementations, the Y planeis extracted from a single one of the decoded subframes-, while in other implementations the resulting Y planeis generated as an aggregate of the Y planes of two or more of the subframes-, for example, as the average of Y values of the corresponding pixels from all four subframes-.

To illustrate,depicts the output of the decoder circuitfor each input bitstream-/decoded subframe-relative to a corresponding 2×2 arrayof a Y plane(representing the original Y plane) and a corresponding 2×2 arrayof a UV plane(representing the original UV plane) of a reconstructed YUV image(representing the YUV image). In this example, the chroma postprocessing circuitextracts a corresponding 2×2 arrayof luminance values (denoted L-L) from the copy of the Y planeobtained from, for example, the decoded subframe(and which corresponds to the 2×2 arrayof) and inserts the 2×2 arrayof luminance values in the corresponding 2×2 arrayof the Y planebeing reconstructed. Further, the chrominance value Cis extracted from the decoded subframeand used to populate the first row/first column position of the arrayof the UV planebeing reconstructed, the chrominance value Cis extracted from the decoded subframeand used to populate the first row/second column position of the array, the chrominance value Cis extracted from the decoded subframeand used to populate the second row/first column position of the array, and the chrominance value Cis extracted from the decoded subframeand used to populate the second row/second column position of the array. In this way, the chroma postprocessing circuitis able to composite the full chroma resolution of the arrayof the reconstructed UV planefrom the subsampled chroma information of each of the four decoded subframes. This same process is applied for each pixel position of the Y planeand the UV plane, resulting in a reconstructed YUV imagehaving the Y planeand the UV planeand with the full chroma resolution of the original YUV image. In the event that subsequent processes require an RGB color space, then at block, the color space conversion circuitcan perform a color space conversion process to convert the YUV imagefrom the YUV color space to an RGB color space, and thereby generating the output image. Otherwise, the YUV imageis output as the output image. In some implementations, color space conversion to RGB may be combined with composition of the full chroma resolution image, resulting in blocksandbeing implemented as a single process.

Thus, with the above-described approach of spreading (the chroma information across multiple subframes such the multiple subframes in aggregate contain the complete chroma resolution following the chroma subsampling process, the video systemcan provide, in effect, 4:4:4 encoding even when one or both of the encoder circuitor the decoder circuitis natively limited to only sub-4:4:4 encoding of any given input stream (that is, configured to apply chroma subsampling to any input stream/subframe as part of the encoding process or is configured to operate only with subsampled chroma information as part of the decoding process). Moreover, because the full Y plane (e.g., Y plane) is represented in each of the subframes, each subframe can be processed by the encoder circuitand the decoder circuitas a separate image.

Moreover, methodfurther can include a performance scaling subprocess where, at block, the decoder circuitor another component of the sink deviceevaluates the workload requirements for decoding the four bitstreams-relative to the current decoding resources available at the sink device. For example, the decoder circuitmay be a software-based decoder that executes on a processor, and the decoder circuitevaluates whether the processor has sufficient available bandwidth or other resources to satisfactorily decode all four bitstreams. In the event that sufficient decoding resources are currently available, then the decoder circuitor other component of the sink deviceconfigures the decoder circuitto decode all four bitstreams. In the event that there are not enough available decoding resources to sufficiently decode all four bitstreams, then the decoder circuitis configured to bypass decoding of one or more of the bitstreams-. In this event, the process of blockremains the same with respect to recovery of the original Y planeas reconstructed Y planesince each encoded substream includes its own separate and full copy of the Y plane. However, bypassing decoding of a bitstream means that the chroma information contained in the bypassed bitstream is unavailable for reconstruction of the UV plane. Accordingly, the compositing process of blockis then modified such that any missing chroma value resulting from bypassed encoding of a bitstream may be recovered via, for example, replication of an adjacent chroma value or by blending, averaging, or otherwise interpolating the missing chroma value from two or more adjacent chroma values.

together illustrate an implementation of the chroma subsampling scheme when three subframes are employed.depicts the encoding process in a similar manner as the example of. As shown, the chroma preprocessing circuitincludes a full copy of the Y planein each of three resulting subframes-, for example, all four luminance values L-Lof any given 2×2 arrayof the Y planeare included in each of the generated subframe-. This may be implemented by, for example, dropping one chroma value in any given 2×2 arrayof the UV planeand then blending the dropped chroma value with one of the other three chroma values to generate a first blended chroma value and blending the dropped chroma value with another of the three chroma values to generate a second blended chroma value, and thus resulting in one original chroma value and two blended chroma values for each 2×2 array. More generally, one of the three subframes is allocated a first subset of the chrominance plane (e.g., every UV value located at the top, leftmost position of every 2×2 array within the UV plane, the second subframe is allocated a first set of blended chrominance values in which each blended chrominance value is generated from the blending of a chrominance value of a second subset of the UV plane with a corresponding chrominance value of a third subset of the UV plane, and the third subframe is allocated a second set of blended chrominance values in which each blended chrominance value is generated from the blending of a chrominance value of a fourth subset of the UV plane with a corresponding chrominance value of the second subset of the UV plane.

The chroma preprocessing circuitthen distributes each of the resulting three chroma values for each 2×2 array—the original chroma value and the two blended chroma values—between the three subframes-to generate three subframes, each of which is subjected to the chroma subsampling processat the encoder circuit. In the example of, with respect to the 2×2 array, the chroma value Cis dropped, and this results in a subsampled subframethat includes the chroma value C, a subsampled subframethat includes a blended chroma value CBthat is an average or other blend of chroma values Cand C, and a subsampled subframethat includes a blended chroma value CBthat is an average or other blend of chroma values Cand C. As with the four subframe process described above, these three subsampled subframes-are subjected to the encoding processby the encoder circuitto generate bitstreams-, which are transmitted to the sink device.

depicts the decoding process of the bitstreams-in a similar manner as the decoding example of. The three bitstreams-are received at the sink device, whereupon they are decoded by the decoder circuitto generate three corresponding decoded subframes-. The chroma postprocessing circuitthen extracts the full copy of the Y planefrom one of the decoded subframes for use as a Y planeof a recovered YUV image, and then composites a UV planeof the recovered YUV imagefrom the UV data included in all three decoded subframes-. For example, for a given 2×2 arrayof the Y plane, the chroma postprocessing circuitextracts a corresponding 2×2 arrayfrom the copy of the Y planeincluded in the decoded subframe. For a given 2×2 arrayof the UV plane, the chroma postprocessing circuitextracts the chroma value Cfrom the decoded subframefor insertion at the first row/first column of the array, extracts the chroma value CBfrom the decoded subframefor insertion at the first row/second column of the array, and extracts the chroma value CBfrom the decoded subframefor insertion at the second row/first column of the array. For the second row/second column of the arraycorresponding to the position of the dropped chroma value C, the chroma postprocessing circuitcan, for example, average or otherwise blend the blended chroma value CBwith the blended chroma value CBto generate a third blended chroma value CB, which is inserted at the second row/second column of the array. This same process is repeated for the remaining 2×2 arrays of the UV plane. Under this approach, a resolution of better than YUV 4:2:2 can be obtained for the output image even when limited to an encoder circuit or decoder circuit that provides only native support for YUV 4:2:2.

together illustrate another implementation of the chroma subsampling scheme when two subframes are employed.depicts the encoding process in a similar manner as the example of. As shown, the chroma preprocessing circuitincludes a full copy of the Y planein each of two resulting subsampled subframesand(subsampled subframesand), for example, all four luminance values L-Lof any given 2×2 arrayof the Y planeare included as copiesin both of the two generated subframesand. However, as there are only two subsampled subframes per image, only two UV values may be provided for any given 2×2 arrayof the UV plane. In the illustrated example, this is implemented by, for example, blending each pair of chroma values in any given 2×2 arrayof the UV planeto generate two corresponding blended chroma values, each of which may be included in one of the corresponding subsample subframes. To illustrate, in the example of, the chroma values Cand Cof arraymay be averaged or otherwise blended to generate a blended chroma value CBfor inclusion in the subsampled subframeand the chroma values Cand Cof arraymay be averaged or otherwise blended to generate a blended chroma value CBfor inclusion in the subsampled subframe. The two subsampled subframesandare then subjected to the encoding processby the encoder circuitto generate bitstreamsand, which are transmitted to the sink device.

depicts the decoding process of the bitstreamsandin a similar manner as the decoding example of. The two bitstreamsandare received at the sink device, whereupon they are decoded by the decoder circuitto generate two corresponding decoded subframesand. The chroma postprocessing circuitthen extracts the full copy of the Y planefrom one of the decoded subframes for use as a Y planeof a recovered YUV image, and then composites a UV planeof the recovered YUV imagefrom the UV data included in both decoded subframesand. For example, for a given 2×2 arrayof the Y plane, the chroma postprocessing circuitextracts a corresponding 2×2 arrayfrom the copy of the Y planeincluded in the decoded subframe. For a given 2×2 arrayof the UV plane, the chroma postprocessing circuitextracts the chroma value CBfrom the decoded subframefor insertion at both the first row/first column and first row/second column of the array, extracts the chroma value CBfrom the decoded subframefor insertion at both the second row/first column and second row/second column of the array. In this way, the resulting YUV imagehas greater chroma resolution than otherwise would be afforded by a conventional YUV 4:2:0 encoding, without requiring that both the encoder circuitand decoder circuitnatively support higher than YUV 4:2:0 encoding.

illustrates another implementation of the chroma subsampling scheme in which chroma information from a previous image may be recovered for use in compositing the chroma information for the present image, referred to herein as a temporal subsampling mode. In this approach, rather than distributing the entire chroma resolution across four subframes for each input image, the chroma preprocessing circuitand the chroma subsampling processinstead distribute half of the chroma information for an image across two subsampled subframes. The encoder circuitthen encodes the resulting subsampled subframes to generate two bitstreams, which are then transmitted to the sink device. At the sink device, the decoder circuitdecodes the two bitstreams to generate two decoded subframes. The chroma postprocessing circuitthen composites half of the chroma information for a resulting output image from the chroma information contained in the two decoded subframes, and buffers this chroma information as well. For the remaining half of the chroma information for the output image, the chroma postprocessing circuitutilizes the chroma information buffered for the preceding input image in the video stream. That is, to reconstruct a full chroma resolution for an image N+1, the chroma postprocessing circuituses the chroma information from the two decoded subframes for image N+1 as well as the chroma information from the two decoded subframes for the preceding image N.

To illustrate, assume a video stream includes a sequence of images N−1 (not shown), N (image), and N+1 (image). For image N, the chroma preprocessing circuitand chroma subsampling processoperate (block) to generate two subsampled subframesand, each having the full luminance resolution of the image N (e.g., luminance values L, L, L, and Lfor the illustrated 2×2 pixel block). The subsampled subframehas only one-fourth of the chrominance information of the image N (e.g., chrominance value Cfor the illustrated 2×2 pixel block) and the subsampled subframehas only a different one-fourth of the chrominance information of the image N (e.g., chrominance value Cfor the illustrated 2×2 pixel blocks). The other half of the chrominance information for image Nis excluded from the resulting subsampled subframesand. The encoder circuitencodes the subframesandand the resulting bitstreams are transmitted to the sink device. This same process was applied to image N−1, resulting in two subframes likewise being encoded and the resulting bitstreams being transmitted to the sink device.

For image N+1, the chroma preprocessing circuitand chroma subsampling processoperate (block) to generate two subsampled subframesand, each having the full luminance resolution of the image N+1 (e.g., luminance values L, L, L, and Lfor the illustrated 2×2 pixel block). The subsampled subframehas only one-fourth of the chrominance information of the image N+1 (e.g., chrominance value Cfor the illustrated 2×2 pixel block) and the subsampled subframehas only a different one-fourth of the chrominance information of the image N+1 (e.g., chrominance value Cfor the illustrated 2×2 pixel blocks). The other half of the chrominance information for image N+1 is excluded from the resulting subsampled subframesand. The encoder circuitencodes the subframesandand the resulting bitstreams are transmitted to the sink device.

This same process was applied to image N−1, resulting in two bitstreams representing two subframes being transmitted to the sink device, with one subframe containing one-fourth of the chrominance information for image N−1 and the other subframe containing another one-fourth of the chrominance information for image N−1.

At the sink device, the encoded bitstreams are decoded to generate two corresponding decoded subframes. The chroma postprocessing circuitoperates (block) to extract the luminance information to generate the luminance plane for a resulting output image(representing image N) and extracts the chrominance information from each of the decoded subframes to composite one-half of the chrominance plane for the output image(as illustrated by chrominance values Cand Cfor the illustrated 2×2 pixel block). The chroma postprocessing circuitalso buffers the chrominance values Cand Cfor use in reconstructing image N+1. For the missing half of the chrominance resolution for output image, the chroma postprocessing circuitutilizes the chrominance information buffered from the decoding of the two subframes received for image N−1 (as represented by chrominance values CX and CY for the illustrated 2×2 pixel block).

For recovering image N+1, the decoder circuitdecodes the corresponding bitstreams to generate two decoded subframes. The chroma postprocessing circuitoperates (block) to extract the luminance information to generate the luminance plane for a resulting output image(representing image N+) and extracts the chrominance information from each of the decoded subframes to composite one-half of the chrominance plane for the output image(as illustrated by chrominance values Cand Cfor the illustrated 2×2 pixel block). The chroma postprocessing circuitalso buffers the chrominance values Cand Cfor use in reconstructing image N+2 (not shown). For the missing half of the chrominance resolution for output image, the chroma postprocessing circuitutilizes the chrominance information buffered from the decoding of the two subframes received for image N (as represented by chrominance values Cand Cfor the illustrated 2×2 pixel block).

Thus, as illustrated, the chrominance information used to composite a chroma plane for a reconstructed image be temporally spread over the transmitted encoded subframes for two images. To ensure that the chrominance information for each subframe is correctly spatially sampled in order to allow reconstruction, the sampling position for each successive subframe may be selected on a N modulo 4 approach. For example, for a 2×2 block, the chrominance information at the top-left pixel for image Nis selected for the first subframe for image N, the chrominance information at the top-right pixel for image N is selected for the second subframe for image N, the chrominance information at the bottom left pixel for image N+1 is selected for the first subframe for image N+1, the chrominance information at the bottom right pixel for image N+1 is selected for the second subframe for image N+1, the chrominance information at the top-left pixel for image N+2 is selected for the first subframe for image N+2, the chrominance information at the top-right pixel for image N+2 is selected for the second subframe for image N+2, and so forth.

Althoughillustrates a particular implementation in which the chroma information is distributed over two subframes, a similar approach can be employed to distribute the chroma information over four subframes instead, such as when further reduction in codec and transmission channel bandwidth is appropriate. In this case only one subframe, containing chrominance value Cwould be sent for image N, one subframe containing chrominance value Cwould be sent for image N+1, one subframe containing chrominance value Cwould be sent for image N+2 and one subframe containing chrominance value Cwould be sent for image N+3. This would result in no increase of the required codec (encoder and decoder) bandwidth compared to regular YUV4:2:0 encoding.

illustrates a methodfor dynamic adaptation of the YUV encoding mode employed in the video systembased on current conditions in accordance with some embodiments. As noted above, the chroma subsampling process can employ two, three, or four subframes depending on the capabilities of the encoder circuitand the decoder circuit, and in some instances may utilize temporal subsampling as described above with reference to. In some implementations, these capabilities, as well as the capabilities of the transmission mediumand other components of the source deviceand sink devicemay change over time. Methodillustrates a technique for dynamically adjusting the granularity or resolution of the YUV encoding mode to best suit the current conditions of the video system.

Accordingly, at blockthe source devicemonitors the current conditions of the video system. These current conditions can include the current available processing resources of one or both of the source deviceand the sink deviceas they may impact the encoding process and decoding process, respectively, as well as the current conditions of the transmission mediumas it impacts the bandwidth or throughput of the transmission mediumfor transmitting encoded data between the source deviceand the sink device. The processing resources may include, for example, available processor resources, available battery resources, work job queue size, current processing latency, and the like. User preferences or other selection criteria, such as QoS requirements, likewise may be identified at block.

At block, the source deviceselects from a plurality of YUV encoding modes the YUV encoding mode that appears to best suit the current conditions, preferences, and/or requirements determined at blockand then configures the chroma preprocessing circuitand encoder circuitto implement the selected mode, as well as signals the sink deviceto configure the chroma postprocessing circuitand decoder circuitto implement the selected mode for an upcoming period (e.g., a specified number of images in the video sequence, a specified time period, or until current conditions sufficiently change to trigger a mode change). For example, these YUV encoding modes can include the four subchannel chroma subsampling mode described above with reference to(block), the three subchannel chroma subsampling mode described above with reference to(block), the two subchannel chroma subsampling mode described above with reference to(block), the temporal chroma subsampling mode described above with reference to(block), as well as a conventional upsampling mode (block). In the conventional upsampling mode, the original UV plane is upsampled such that when it is chroma subsampled, the resulting subsampled UV plane has the original chroma resolution. However, while retaining the original chroma resolution in one stream, the conventional upsampling mode often is at risk of being unsupported by certain implementations due to their maximum resolution thresholds—that is, the UV plane cannot be sufficiently upsampled without exceeding a maximum image size set for the encoder. Moreover, the conventional upsampling mode lacks robustness as the sink devicecannot disregard the upsampled-resolution stream if they cannot decode it—that is, there is no fallback to recover a lower resolution version at the sink devicein the event that the upsampled version exceeds the sink device's capabilities.

To illustrate an operation of method, assume the current capabilities indicate that the source devicepresently has resources for encoding at a high resolution, the decoder has resources for decoding at a high resolution, and the current bandwidth of the transmission mediumis in excess of what is needed for any mode. In such an instance, the source devicemay elect to implement either the conventional upsampling mode (block) or implement the four subchannel mode (block). In a similar situation but with an indication that either the encoder circuitor the decoder circuitare incapable of supporting the high resolution required for the conventional upscaling mode, then the four subchannel mode may be implemented for the upcoming period. However, should one or more of the source device, sink device, or transmission mediumbe moderately resource-limited, then the source devicemay signal a dynamic degradation to the three-channel mode (block), which still provides greater than YUV 4:2:2 resolution. Further, should one or more of the source device, sink device, or transmission mediumreport extensive limitations on resource availability, then the source devicemay signal a dynamic degradation to the two-channel mode (block) so as to conserve processing or transmission resources, while still providing higher than YUV 4:2:0 resolution.

illustrates an example hardware implementation of a processing systemthat may be implemented at one or both of the source deviceor sink devicein accordance with some embodiments. The processing systemincludes one or more processors, such as a central processing unit (CPU), one or more co-processors(or other hardware accelerators), such as a graphics processing unit (GPU), one or more memories(e.g., system memory, cache, hard disk drive, etc.), one or more network interfaces(e.g., a WLAN interface, an Ethernet interface, etc.), one or more input/output (I/O) devices, such as a display, a keyboard, a touchscreen, and the like, and an encoder/decoder (codec), which represents the encoder circuitin an implementation of the processing systemas the source deviceor represents the decoder circuitin an implementation of the processing systemas the sink device. The codecmay be implemented as a hardware-based codec, such as an ASIC or hard-coded logic in a hardware accelerator, as a software-based codec, such as via software stored in the memoryand executed by one or both of the processoror GPU, or a combination thereof.

The one or more memoriesstore one or more sets of executable code that, when executed, cause the one or more processorsand/or the one or more co-processorsto perform certain actions described above. To illustrate, in conjunction with execution of software instructions providing for an operating system (OS) and associated drivers, the processorand/or co-processormay execute subsampling softwarestored in the memoryand which cause the processorand/or co-processorto implement the operation of the chroma preprocessing circuitor implement the operation of the chroma postprocessing circuit. For example, the subsampling softwaremay be implemented as part of an application programming interface (API) or other interface layer between the OS and the codec, and thus serving to provide YUV 4:4:4 support for a codecthat only natively supports sub-4:4:4 encoding/decoding, using the techniques described above.

In some embodiments, the apparatus and techniques described above are implemented in a system including one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips), such as the source deviceor sink devicedescribed above with reference to. Electronic design automation (EDA) and computer aided design (CAD) software tools may be used in the design and fabrication of these IC devices. These design tools typically are represented as one or more software programs. The one or more software programs include code executable by a computer system to manipulate the computer system to operate on code representative of circuitry of one or more IC devices so as to perform at least a portion of a process to design or adapt a manufacturing system to fabricate the circuitry. This code can include instructions, data, or a combination of instructions and data. The software instructions representing a design tool or fabrication tool typically are stored in a computer readable storage medium accessible to the computing system. Likewise, the code representative of one or more phases of the design or fabrication of an IC device may be stored in and accessed from the same computer readable storage medium or a different computer readable storage medium.

A computer readable storage medium may include any non-transitory storage medium, or combination of non-transitory storage media, accessible by a computer system during use to provide instructions and/or data to the computer system. Such storage media can include, but is not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media (e.g., floppy disc, magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or Flash memory), or microelectromechanical systems (MEMS)-based storage media. The computer readable storage medium may be embedded in the computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., a magnetic hard drive), removably attached to the computing system (e.g., an optical disc or Universal Serial Bus (USB)-based Flash memory), or coupled to the computer system via a wired or wireless network (e.g., network accessible storage (NAS)).

In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer readable storage medium. The software can include the instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer readable storage medium can include, for example, a magnetic or optical disk storage device, solid state storage devices such as Flash memory, a cache, random access memory (RAM) or other non-volatile memory device or devices, and the like. The executable instructions stored on the non-transitory computer readable storage medium may be in source code, assembly language code, object code, or other instruction format that is interpreted or otherwise executable by one or more processors.

Note that not all of the activities or elements described above in the general description are required, that a portion of a specific activity or device may not be required, and that one or more further activities may be performed, or elements included, in addition to those described. Still further, the order in which activities are listed are not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature of any or all the claims. Moreover, the particular embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the claims below. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims below.