An improved method for temporal motion vector prediction for inter block HEVC is provided that relies on a block translational model. The method adds an offset to a temporal motion vector predictor (TMVP) to improve prediction accuracy. The method first designates a current prediction block as an area for motion compensation where all the pixels inside the prediction block perform identical translation temporally using motion vectors MVs. A coordinate offset is then derived for a current prediction block from the MVs of its spatially neighboring blocks. The offset TMVP is then defined for the current prediction block as the MV of an offset block which is in the geometrical location of the current prediction block coordinate plus the coordinate offset in a specified temporal reference picture. The offset TMVP is then used to code MVs.
Legal claims defining the scope of protection, as filed with the USPTO.
designating a current prediction block as an area for motion compensation using HEVC where all the pixels inside the current prediction block perform identical translation temporally using either one or more motion vectors MVs; deriving a coordinate offset for the current prediction block from the MVs of its spatially neighboring blocks; defining an offset of a temporal motion vector predictor (TMVP) for the current prediction block as the MV of an offset block which is in the geometrical location of the current prediction block coordinate plus the coordinate offset in a specified temporal reference picture; and using the offset TMVP to code MVs, wherein the motion vectors of neighboring prediction blocks to the current prediction block are used to calculate the offset for the TMVP, wherein the neighboring prediction blocks located in a first three positions in a merge candidate list for the current prediction block are used in calculating the offset for the TMVP, wherein the three neighboring prediction blocks comprise a left (L), an above (A), and an above-left (AL), dx=median (Lx, ALx, Ax) dy=median (Ly, ALy, Ay) wherein with the three neighboring prediction blocks, the offset for the TMVP for the current prediction block is derived as median of motion vectors of these neighbors, as follows: wherein Lx, ALx, Ax are the x component of motion vectors of Left neighbor, Above-left neighbor, and Above neighbor, respectively, and wherein Ly, ALy, Ay are the y component of Left neighbor, Above-left neighbor, and Above neighbor, respectively. . A method of temporal motion vector prediction for inter block coding in High Efficiency Video Coding (HEVC) that relies on a block based translational model, the method comprising:
claim 1 assuming that the current prediction block is at the position of coordinate (x, y) in the current picture; and (x′, y′) =(x, y) +(dx, dy) =(x+dx, y+dy). adding a coordinate offset of (dx, dy) to the coordinate (x,y) to provide the offset TMVP (x′,y′) as . The method of, wherein in the step of defining the offset TMVP, the offset TMVP is further defined by the steps comprising:
claim 1 . The method of, wherein syntax elements are required in coding a bitstream to indicate values for the offset TMVP.
claim 1 . The method of, wherein one offset is shared for multiple prediction blocks or coded with coarser granularity than the final fractional motion vector accuracy.
claim 1 . The method of, wherein motion vectors of neighboring prediction blocks are normalized to compensate for the difference in temporal distances between references used among the neighboring prediction blocks.
claim 1 providing a TMVP offset mode that can be turned on or off, wherein in the on mode the offset TMVP with the added offset is used to code MVs, and in the off mode a TMVP without the added offset is used to code MVs, wherein an explicit signal is provided for turn on and turn off of the offset TMVP mode using a flag provided at a CU, slice or sequence level. . The method of, wherein the step of using MV prediction with the TMVP to code MVs, comprises:
(a) designating a current prediction block as an area of said video for motion compensation where all pixels defined by the current prediction block perform identical translation temporally using at least one motion vector, designating another prediction block as another area of said video for motion compensation where all pixels defined by the another prediction block perform identical translation temporally using at least one motion vector; (b) deriving a coordinate offset for the current prediction block from at least one motion vector of a set of spatially neighboring blocks relative to the current prediction block; (c) wherein a selection of said set of spatially neighboring blocks is based upon a merge candidate list for said current prediction block; (d) wherein said merge candidate list includes a left neighboring block (L), and an above neighboring block (A), where said merge candidate list includes an ordered list of candidate neighboring blocks, where said left neighboring block (L) is directly left of and aligned with the current prediction block, where said above neighboring block (A) is directly above of and aligned with the current prediction block; (e) wherein said left neighboring block (L), and said above neighboring block (A), are included within two of a first position, a second position, and a third position of the ordered list of candidate neighboring blocks of said merge candidate list, where said left neighboring block (L) is included within one of said first position, the second position, and the third position, where said above neighboring block (A) is included within a different one of said first position, the second position, and the third position than said left neighboring block (L); (f) defining an offset temporal motion vector predictor (TMVP) for the current prediction block as the coordinate offset based upon the at least one motion vector of an offset block selected from the merge candidate list and further based upon another coordinate offset in a specified temporal reference picture that is in a temporally different picture than said current prediction block; and (g) decoding said current prediction block based upon said offset TMVP. 7. A method of temporal motion vector prediction for inter block coding of video within a bitstream that relies on a block based translational model, the method comprising:
Complete technical specification and implementation details from the patent document.
This is a continuation application of 17/487,450 (U.S. RE50,091 E) and a reissue of 15/726,078 (U.S. 10,432,960 B2). 17/487,450 is a reissue of 15/726,078 IUS 10,432,960 B2) and 15/726,078 claims the benefit of provisional 62/404,573. The entire disclosures of the above applications are hereby incorporated by reference herein for all purposes.
Technical Field
The present invention relates to coding using High Efficiency Video Coding (HEVC or H-265). More particularly, the present system relates to motion vector prediction for coding of HEVC blocks.
Related Art
State of the art video quality is achieved today by performing complex High Efficiency Video Coding (HEVC) video encoding with the video pictures broken into macroblocks. Motion vector determination enables more efficient coding of the blocks making up a picture. The HEVC coding standard (also called H.265) is a coding standard promulgated by the ISO/IEC MPEG standardization organizations. HEVC supports resolutions higher than “high definition,” which means pixels may be represented by a larger number of bits than the high definition pictures. For example, 4K resolutions may include images that are 4,000 pixels wide compared to high definition images that are 1920 pixels wide.
Temporal motion prediction is an effective method to increase the coding efficiency and provides high compression necessary for HEVC. HEVC uses a translational model for temporal motion prediction. According to the translational model, a prediction signal for a given current unit in a current picture is generated from a corresponding reference unit in a reference picture. The coordinates of the reference unit are given by a motion vector that describes the translational motion along horizontal (x) and vertical (y) directions that would be added/subtracted to/from the coordinates of the current unit. A decoder needs the motion vector to decode the compressed video.
HEVC relies on a block based translational model for its temporal prediction (inter coding) in order to keep complexity and overhead bits low. For inter coding block, HEVC designates a prediction block as an area for motion compensation where all the pixels inside the prediction block performs identical translation temporally using either one or two motion vectors (MV). Motion vector prediction is used to code motion vector(s) to reduce the overhead bits for motion vector signaling. Motion estimation or prediction is a process of determining a motion vector (MV) for a current unit of video. The motion estimation process searches for a best match prediction for a current unit block of video (e.g., a prediction block) over reference pictures. For a current inter block, its motion vector predictor can be derived from the motion vectors of its spatially neighboring blocks and/or the temporally collocated block.
It is desirable to provide improvements in motion vector prediction accuracy to make HEVC coding more efficient.
Embodiments of the invention provide a method for temporal motion vector prediction for inter block HEVC that relies on a block translational model, enabling more efficient encoding. The method adjusts the coordinate of a temporal motion vector predictor (TMVP) by adding an offset to improve prediction accuracy. The modified coordinate provides better prediction when there is significant object motion between the frames
In particular, the method begins by designating a current prediction block as an area for motion compensation using HEVC where all the pixels inside the prediction block perform identical translation temporally using either one or more motion vectors MVs. A coordinate offset is derived for a current prediction block from the MVs of its spatially neighboring blocks. An offset TMVP is then defined for the current prediction block as the MV of an offset block which is in the geometrical location of the current prediction block coordinate plus the coordinate offset in a specified temporal reference picture. The offset TMVP can be used to code MVs to reduce the overhead bits for motion vector signaling. Additionally, the offset TMVP can be used in motion vector predictor calculation of merge mode such as alternative temporal motion vector prediction (ATMVP), or spatial temporal motion vector prediction (STMVP).
The offset TMVP in one embodiment is specifically defined assuming that the current prediction block is at the position of coordinate (x, y) in the current picture. the coordinate offset is added by doing the following:
adding a coordinate offset of (dx, dy) to the coordinate (x,y) to give the offset TMVP as(x′,y′)=(x,y)+(dx,dy)=(x+dx,y+dy).
In an explicit approach for encoding using the TMVP offset according to embodiments of the present invention, syntax elements expressing the offset are used. One offset can be shared for multiple prediction blocks.
In an implicit approach for encoding using the TMVP offset according to embodiments of the present invention, motion vectors of neighboring prediction blocks to the current prediction block are used to calculate the offset for the TMVP. In one example, the neighboring prediction blocks located in a first three positions in a merge candidate list for the current prediction block are used in calculating the TMVP offset. In another example, the three neighboring prediction blocks, the left (L), the above (A), and the above-left (AL), are used for computing the TMVP offset. One possible example for calculating the TMVP offset for the current prediction block is to use a median of motion vectors of these neighbors, as follows:dx=median(Lx,ALx,Ax)dy=median(Ly,ALy,Ay)
wherein Lx, ALx, Ax are the x component of motion vectors of Left neighbor, Above-left neighbor, and Above neighbor, respectively, and
wherein Ly, ALy, Ay are the y component of Left neighbor, Above-left neighbor, and Above neighbor, respectively.
An added TMVP offset mode can be used in one embodiment of the present invention that can be turned on and off with either the explicit or implicit means. Implicit signaling for turn on and turn off of the offset TMVP mode can be based on coding information of neighboring blocks. Explicit signaling for turn on and turn off in the offset TMVP mode can be thru a flag at a CU, slice or sequence level.
1 FIG. 102 104 102 104 102 104 102 104 shows a simplified system for encoding and decoding video according to embodiments of the present invention. System includes an encoderand a decoder. Encoderand decodermay use a video coding standard to encode and decode video, such as HEVC. Specifically, encoderand decodermay use syntax elements from the HEVC range extension. Also, other elements of encoderand decodermay be appreciated.
102 104 Encoderand decoderperform temporal prediction through motion estimation and motion compensation. Motion estimation is a process of determining a motion vector (MV) for a current unit of video. For example, the motion estimation process searches for a best match prediction for a current unit block of video (e.g., a prediction block) over reference pictures. The best match prediction is described by the motion vector and associated reference picture ID. Also, a reference unit in a B picture may have up to two motion vectors that point to a previous reference unit in a previous picture and a subsequent reference unit in a subsequent reference picture in the picture order. Motion compensation is then performed by subtracting a reference unit pointed to by the motion vector from the current unit of video. In the case of bi-prediction, the two motion vectors point to two reference units, which can be combined to form a combined bi-directional reference unit.
102 104 104 1 104 2 104 1 104 2 To perform motion estimation and compensation, encoderand decoderinclude motion estimation and compensation blocks-and-, respectively. For bi-directional prediction, the motion estimation and compensation blocks-and-can use a combined bi-directional reference unit in the motion compensation process for the current unit. Syntax elements are further used in the motion prediction process.
102 104 102 104 1 FIG. For the encoderand decoderof, embodiments of the present invention contemplate that software to enable them to perform functions described to follow for the present invention is provided in a memory. The encoderand decoderare further contemplated to include one or more processors that function in response to executable code stored in the memory to cause the processor to perform the functions described.
2 FIG. 200 1 200 5 200 3 202 1 202 1 200 1 200 5 200 1 202 2 200 5 202 3 202 1 illustrates video pictures with reference unit blocks for motion estimation and compensation. The video includes a number of pictures---. A current picture is shown at-and includes a current unit of video-. Current unit-may be bi-predicted using reference unit blocks from reference pictures in other pictures, such as a previous picture-in the picture order and a subsequent picture-in the picture order. Picture-includes a reference unit-and picture-includes a reference unit block-, both of which can be used to predict current unit block-.
104 1 202 2 202 3 202 1 104 1 202 1 102 104 Once the current picture is established and the reference unit blocks are determined, motion estimation and compensation block-can determine motion vectors that represent the location of reference unit blocks-and-with respect to current unit block-. Then, motion estimation and compensation block-calculates a difference between the combined reference unit block and the current unit block-. Encoderoutputs the motion vectors in an encoded bitstream that is sent to decoder.
104 104 202 2 202 3 202 1 104 202 1 104 104 2 202 1 104 2 202 2 202 3 202 1 Decoderreceives the encoded bitstream and can reconstruct the pictures of the video. Decodermay reconstruct reference unit blocks-and-from the encoded bitstream prior to decoding current unit block-. Also, decoderdecodes the motion vectors for current unit block-. Then, in decoder, motion estimation and compensation block-are used to reconstruct the current unit block-. The motion estimation and compensation block-uses the motion vectors to locate reconstructed reference unit blocks-and-and reconstruct the current unit block-.
Motion vector prediction is used in motion vector coding process to exploit correlation between the coding motion vector and its selected predictor. Due to the characteristics of natural video, object generally moves in a smooth, linear trajectory from frame to frame. This behavior makes the motion vector of the temporally collocated block as a powerful motion vector predictor for a current block, and it is hence used in HEVC motion vector coding.
In HEVC, for a current prediction block in a current picture, the motion vector of its temporal collocated prediction block, which is in the same geometrical location in a specified temporal reference picture as the current prediction block in the current picture, is defined as the temporal motion vector predictor (TMVP) for the current prediction block. Specifically, the collocated block has the same spatial coordinate (x,y) in the reference picture as the current prediction block (x,y) in the current picture. The collocated position can however be suboptimal when there is significant object motion between the frames. In such case, the collocated position may represent a different object and its motion vector is not a useful TMVP.
Accordingly, embodiments of the present invention introduce a way to improve TMVP effectiveness, especially when there are a lot of movements between the frames. Instead of using the same coordinate in the reference picture as the coding prediction block in the current picture, embodiments of the present invention add a coordinate offset to the coordinate for the TMVP location.
3 FIG. 300 302 304 302 306 is a flowchart showing steps of a method of adding an offset to a TMVP according to embodiments of the present invention. The method uses a block translational model and begins in stepwith designating a current prediction block as an area for motion compensation where all the pixels inside the prediction block perform identical translation temporally using either one or more motion vectors (MVs). In the next step, a coordinate offset is derived for the current prediction block from the MVs of its spatially neighboring blocks. In step, the offset TMVP is defined for the current prediction block as the MV of an offset block which is in the geometrical location of the current prediction block coordinate plus a coordinate offset computed in stepin a specified temporal reference picture. Finally, in stepthe MV prediction with the offset TMVP is used to code MVs.
4 FIG. 400 402 is a flowchart showing specifics of the steps to further define the offset TMVP. First in step, an assumption is made that the current prediction block is at the position of coordinate (x, y) in the current picture. Next, in stepa coordinate offset of (dx, dy) is added to the coordinate (x,y) to define the offset TMVP (x′, y′) as follows:(x′,y′)=(x,y)+(dx,dy)=(x+dx,y+dy).
The TMVP offset (dx, dy) can be determined explicitly or implicitly. Details of the two approaches are described to follow.
For explicit approach, syntax elements in coding bitstream can be used to indicate TMVP offset values. To reduce the overhead bits, one offset may be shared for multiple prediction blocks and coded with coarser granularity than the final fractional motion vector accuracy.
For implicit approach, a motion vector derivation method is specified so that decoder can repeat the same process and be able to regenerate the same TMVP offset. In this simplified approach, the motion vectors of neighboring prediction blocks are used to calculate the offset for TMVP. Motion vectors of neighboring prediction blocks in this approach are normalized to compensate for the difference in temporal distances between references used among these prediction blocks.
5 FIG. 6 FIG. 5 FIG. 6 is a flowchart showing calculation of the coordinate offset using the neighboring prediction blocks.illustrates the three neighboring prediction blocks used to calculate the coordinate offset for the current block (C) in. The prediction blocks in FIG.include the above left (AL), above (A), above right (AR), left (L), right (R), below left (BL), below (B), and below right (AR).
500 601 602 603 502 5 FIG. In a first stepof, the three neighboring prediction blocks are used for the current prediction block to calculate the offset TMVP. The three neighboring prediction blocks include the left (L), the above (A), and the above-left (AL). Next, in step, the current prediction block is derived as a median of motion vectors of these neighbors, as follows:dx=median(Lx,ALx,Ax)dy=median(Ly,ALy,Ay)
wherein Lx, ALx, Ax are the x component of motion vectors of Left neighbor, Above-left neighbor, and Above neighbor, respectively, and
wherein Ly, ALy, Ay are the y component of Left neighbor, Above-left neighbor, and Above neighbor, respectively.
An added TMVP offset mode can be used in one embodiment of the present invention that can be turned on and off with either the explicit or implicit means. Implicit signaling for turn on and turn off of the offset TMVP mode can be based on coding information of neighboring blocks. Explicit signaling for turn on and turn off in the offset TMVP mode can be thru a flag at a CU, slice or sequence level.
Although the present invention has been described above with particularity, this was merely to teach one of ordinary skill in the art how to make and use the invention. Many additional modifications will fall within the scope of the invention as that scope is defined by the following claims.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
June 26, 2024
June 23, 2026
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.