Deep Learning-Based 2d Gaussian Estimation Method and Apparatus Thereof

This application claims the benefit of priority under 35 U.S.C. § 119(a) to Korean Patent Application No. 10-2024-0186375 filed on Dec. 13, 2024, and Korean Patent Application No. 10-2025-0044717 filed on Apr. 7, 2025, the entire contents of which is incorporated herein by reference.

The present disclosure relates to a deep learning-based 2D Gaussian estimation method and apparatus thereof.

Conventionally, in order to detect specific patterns or regions in an image, traditional image processing algorithms (e.g., Canny edge detection, Hough transform) or local feature-based keypoint detection techniques (such as SIFT and SURF) have mainly been used. These algorithms are effective in detecting feature points in an image and utilizing them to identify specific patterns or structures. However, such conventional methods are not suitable for directly extracting a mathematical representation of an image (e.g., a Gaussian distribution), and require complex post-processing.

Recently, techniques for learning high-dimensional information of images using deep learning have attracted attention. In particular, CNN-based models have shown outstanding performance in object detection and segmentation; however, these techniques are mainly focused on classification and bounding-box extraction, and therefore are difficult to apply to extraction of a mathematical representation of an image, such as a Gaussian distribution.

As prior art, there exist spatial partitioning methods using Voronoi diagrams and techniques for predicting object centers using a gradient map and a center map. However, there is no technique that directly extracts 2D Gaussians by combining such methods with deep learning.

The present disclosure is directed to providing a deep learning-based 2D Gaussian estimation method and apparatus thereof.

In addition, the present invention is directed to providing a deep learning-based 2D Gaussian estimation method and an apparatus thereof, which are capable of estimating 2D Gaussians having an appropriate distribution according to the complexity of a given image by training a model using a Voronoi diagram. That is, in a complex region (a region having a high frequency) of an image, a large number of 2D Gaussians can be estimated, whereas in a simple region (a region having a low frequency), a small number of 2D Gaussians can be estimated.

Furthermore, the present invention is directed to providing a deep learning-based 2D Gaussian estimation method and an apparatus thereof, in which 2D Gaussians estimated by the model can be used as initial values of 3D Gaussians, thereby shortening a development and deployment process of a 3DGS (3D Gaussian Splatting)-based system, improving computational efficiency, and enhancing three-dimensional rendering quality.

In addition, the present invention is directed to providing a deep learning-based 2D Gaussian estimation method and an apparatus thereof, which can offer data processing and representation efficiency by mathematically expressing high-dimensional data in a concise form, and which are applicable to various applications such as computer vision, medical image analysis, and robot vision.

According to one aspect of the present disclosure, a deep learning-based 2D Gaussian estimation method is provided.

According to an embodiment of the present disclosure, the deep learning-based 2D Gaussian estimation method, may comprise: (a) randomly generating a Voronoi diagram and partitioning each region of the Voronoi diagram into a respective polygon; (b) configuring a dataset by generating a gradient map, a center probability map, and a covariance map based on each of the partitioned polygons; (c) training a deep learning-based Gaussian estimation model using the dataset so that the deep learning-based Gaussian estimation model is configured to output, for an input image, corresponding center probability and covariance maps; and (d) inputting an image to the trained deep learning-based Gaussian estimation model to generate a center probability map and a covariance map, and deriving 2D Gaussian parameters including a mean and a covariance by post-processing the target center probability map and the target covariance map.

In step (a), the method may comprise: (a1) randomly generating seed points and generating a Voronoi diagram based on the seed points; and (a2) randomly assigning colors to the respective regions of the Voronoi diagram.

A number of seed points, a number of colors, and a distribution of polygons may be randomly determined.

Generating the covariance map of the dataset may comprise: (b1) designating, as a center point, a target interior pixel of each polygon; (b2) calculating a variance and a covariance by using the center point and coordinate values of boundary pixels of the polygon including the target interior pixel; and (b3) generating a covariance matrix for the target interior pixel by using the calculated variance and covariance, wherein steps (b1) to (b3) are repeatedly performed for all interior pixels of each polygon so as to generate the covariance map.

The covariance map may be expressed as any one of: a first covariance map having a size of 2×2×H×W and including all elements of a 2×2 covariance matrix; a second covariance map having a size of 3×H×W and obtained by reducing the 2×2 covariance matrix so as to include only three distinct elements of the 2×2 covariance matrix; and a third covariance map having a size of 3×H×W and comprising components {a, b, c} of a lower triangular matrix obtained by applying Cholesky factorization to a 2×2 covariance matrix corresponding to each pixel constituting the covariance map so as to decompose the 2×2 covariance matrix into a product of the lower triangular matrix and a transpose matrix of the lower triangular matrix.

The deep learning-based Gaussian estimation model may comprise: an encoder configured to receive the image and extract a feature map; a first decoder configured to reconstruct the feature map to generate a gradient map; a weight feature map generation module configured to generate a gradient weight feature map by scaling the gradient map to a size identical to that of the feature map and performing element-wise multiplication; a second decoder configured to receive the gradient weight feature map and generate a center probability map; and a third decoder configured to receive the gradient weight feature map and generate a covariance map.

According to another aspect of the present disclosure, an apparatus for performing the deep learning-based 2D Gaussian estimation method is provided.

According to an embodiment of the present disclosure, a computing device may be provided, the computing device comprising: a dataset construction unit configured to randomly generate a Voronoi diagram and construct a dataset utilizing the Voronoi diagram; a training unit configured to train a deep learning-based Gaussian estimation model to generate a Center Probability Map and a Covariance Map for an input image using the dataset; and a Gaussian estimation unit configured to input an image into the trained deep learning-based Gaussian estimation model to generate a Center Probability Map and a Covariance Map, and obtain 2D Gaussian parameters including a mean and a covariance by post-processing the Center Probability Map and the Covariance Map.

Singular forms used in this specification include plural forms unless the context clearly indicates otherwise. In the specification, the term “configured”, “include”, or the like should not be construed as necessarily including several components or several steps described herein, in which some of the components or steps may not be included or additional components or steps may be further included. Further, the terms “˜unit”, “module”, and the like mean a unit for processing at least one function or operation and may be implemented by hardware or software or by a combination of hardware and software.

Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 FIG. 2 4 FIGS.to 5 FIG. 6 FIG. 7 8 FIGS.and 9 10 FIGS.and 11 12 FIGS.and 13 FIG. 14 FIG. is a flowchart illustrating a deep learning-based adaptive Gaussian estimation method according to an embodiment of the present disclosure,are diagrams illustrating examples of a Voronoi diagram for an input image according to an embodiment of the present disclosure,is a diagram illustrating a Gradient Map generated utilizing a Voronoi diagram according to an embodiment of the present disclosure,is a diagram illustrating a Center Probability Map and a Covariance Map generated utilizing a Voronoi diagram according to an embodiment of the present disclosure,are diagrams illustrating a method of generating a Covariance Map according to an embodiment of the present disclosure,are diagrams illustrating the architecture of a deep learning-based Gaussian estimation model according to an embodiment of the present disclosure,are exemplary diagrams visualizing 2D Gaussians according to an embodiment of the present disclosure,is a diagram illustrating a process of generating a 2D Gaussian according to an embodiment of the present disclosure, andis a diagram illustrating a process of converting a 2D Gaussian into a 3D Gaussian according to an embodiment of the present disclosure.

110 100 In step, a computing devicerandomly generates a Voronoi diagram and partitions each region into a respective polygon.

This will be described in more detail below.

2 3 FIGS.and illustrate an example of a Voronoi diagram.

100 2 FIG. 2 FIG. The computing devicemay generate a plurality of seed points and then partition each region into a respective polygon (see). As shown in, the Voronoi diagram partitions a plane into regions each consisting of a set of points closest to a corresponding seed point, and each partitioned region may be formed as a convex polygon.

100 Accordingly, the computing devicemay randomly generate seed points. A number of the seed points may be randomly generated in a range of 10 to 500.

3 FIG. 100 Subsequently, as illustrated in, the computing devicemay assign a respective color to each polygon.

100 100 The computing devicemay randomly determine a number of colors. For example, the computing devicemay randomly generate 10 to 100 colors, and randomly assign and fill the partitioned polygonal regions with colors within the generated color range. In this case, a number of color clusters may be randomly set to be in a range of 1 to 10.

In this manner, by preliminarily generating a predetermined number of colors and then assigning selected colors to the polygonal regions such that adjacent regions are assigned the same color, cases in which the regions appear as concave shapes in the input image can be generated.

4 FIG. 4 FIG. 100 In addition, as illustrated in, a distribution of polygons may also be randomly determined. As shown in, the computing devicemay generate the Voronoi diagram with various distributions.

100 In summary, the computing devicemay randomly generate the Voronoi diagram by randomly adjusting (determining) a number of seed points, a number of colors, and a polygon distribution.

115 100 In step, the computing deviceconfigures a dataset by using the Voronoi diagram.

This will be described in more detail below.

100 The computing devicemay generate a gradient map and a center probability map based on each of the partitioned polygons.

100 100 5 FIG. 6 FIG. 6 a FIG.() For example, the computing devicemay generate the gradient map on the basis of boundary lines of each polygon (see). In addition, the computing devicemay generate the center probability map by using a centroid of each polygon (see). As illustrated in, the center probability map is a probability map that encodes, for each pixel, a likelihood that the pixel is a center of a polygon. For each pixel within a polygon, a Euclidean distance to the center is normalized by a minimum edge distance, and is adjusted by a fast decaying function. This can be expressed mathematically as in Equation 1.

Here, d denotes a normalized distance to a centroid.

100 6 b FIG.() In addition, the computing devicemay generate a covariance map using centroid information of each polygon. A result of visualizing sampled center regions of Gaussians from the covariance map is illustrated in.

7 FIG. For ease of understanding and explanation, a description will now be given with reference to.

7 a FIG.() 7 b FIG.() 7 b FIG.() 7 c FIG.() 7 c FIG.() 710 100 As shown in, a target polygonhas six vertices, and when approximated, it may be represented as shown in. It is assumed that further approximatingyields. With reference to, the computing devicemay set each interior pixel of the polygon as a center point (mean value).

730 730 100 730 For convenience of description, such an interior pixel will be referred to as a target interior pixel. The target interior pixelmay be set as the center point (mean value). Subsequently, the computing devicemay calculate a variance and a covariance by using the center pointand coordinate values of boundary pixels.

100 730 That is, the computing devicemay calculate a variance of the target interior pixelas in Equation 2, and may calculate a covariance as in Equation 3.

i i x y In Equation 2 and Equation 3, i denotes an index of the boundary pixels, xdenotes an x-coordinate value of the boundary pixels, anddenotes an x-coordinate value of the center point. In addition, ydenotes a y-coordinate value of the boundary pixels, anddenotes a y-coordinate value of the center point.

100 730 8 FIG. In this manner, when the variance and covariance are respectively calculated by using the coordinates of the center point and the boundary pixels, the computing devicemay generate a 2×2 covariance matrix for the target interior pixel. By repeating this process, 2×2 covariance matrices may be generated for all pixels inside the polygon (see).

The 2×2 covariance matrix may be expressed as in Equation (4).

By repeatedly performing the above process, a four-dimensional covariance map of size 2×2×H×W may be generated.

As can be seen from Equation 4, since the covariance matrix is a 2×2 symmetric matrix, the off-diagonal elements Cov(x, y) and Cov(x, y) are identical. Accordingly, although a four-dimensional covariance map of size 2×2×H×W is generated, by using this symmetry only three entries have mutually different values, and thus the covariance map can practically be represented as a three-dimensional covariance map of size 3×H×W.

100 x y In order to generate the covariance map, a 2×2 covariance matrix has to be calculated for all pixels, including interior pixels and boundary pixels of the polygon. Accordingly, the computing devicemay set each pixel as a center point coordinates ((,)), and may compute a covariance matrix by using center coordinates

of boundary pixels constituting σ a boundary of the polygon.

100 The computing devicemay configure a gradient map, a center probability map, and a covariance map for an input image as a dataset.

100 According to an embodiment of the present invention, the computing devicemay apply Cholesky factorization to the covariance map. Accordingly, a Cholesky-factorized covariance map can be represented as in Equation 5.

100 The computing devicemay apply Cholesky factorization to a covariance matrix to decompose a 2×2 covariance matrix corresponding to each pixel of the covariance map into a product of a lower triangular matrix and a transpose matrix of the lower triangular matrix, thereby representing a covariance for each pixel with three parameters {a, b, c}.

120 100 In step, the computing devicemay train a deep learning-based Gaussian estimation model using the dataset so that the deep learning-based Gaussian estimation model generates a center probability map and a covariance map for an input image.

9 10 FIGS.and An overall architecture of the deep learning-based Gaussian estimation model is illustrated in.

910 920 930 940 950 The deep learning-based Gaussian estimation model includes an encoder, a first decoder, a weighted feature map generation module, a second decoder, and a third decoder.

910 910 The encoderis a means for receiving an input image and extracting a feature map. The encodermay be a neural-network-based model or a feature pyramid network-based model.

910 For example, the encodermay be a model based on ResNet-50.

920 910 The first decoderis a means for reconstructing the feature map extracted by the encoderand generating a gradient map. The gradient map serves as an auxiliary output and plays a role in enhancing feature representation through weighting before predicting final outputs, namely a center probability map and a covariance map. Accordingly, the gradient map may be included in the dataset.

920 The first decodermay be trained so that a difference between the generated gradient map and a gradient map included in the dataset is minimized.

930 The weighted feature map generation modulemay scale the gradient map to have the same size as the feature map, and may generate a gradient-weighted feature map by performing element-wise multiplication on the scaled gradient map and the feature map.

940 950 940 950 The gradient-weighted feature map is respectively input to the second decoderand the third decoder. The second decodermay predict a center probability map by using the gradient-weighted feature map, and the third decodermay predict a covariance map by using the gradient-weighted feature map.

940 950 The second decoderand the third decodermay also be trained by using the center probability map and the covariance map included in the dataset.

100 In this manner, the computing devicemay train the deep learning-based Gaussian estimation model so that the deep learning-based Gaussian estimation model generates a center probability map and a covariance map for an input image.

The deep learning-based Gaussian estimation model may be trained by using a mean squared error (MSE) that minimizes differences between the center probability map and the covariance map included in the dataset and a center probability map and a covariance map output by the deep learning-based Gaussian estimation model.

ij ij ij ij ij ij Assume that C, Σ, and Gdenote datasets representing a center probability map, a covariance map, and a gradient map at a pixel (i, j), respectively, and that Ĉ, {circumflex over (Σ)}, and Ĝdenote a center probability map, a covariance map, and a gradient map predicted by the deep learning-based Gaussian estimation model. In the spatial dimension H×W, three loss functions can be expressed as in Equation 6.

A final loss function may be calculated as in Equation 7 by applying an edge-aware technique utilizing the gradient map while balancing center prediction and covariance prediction.

C Σ G 125 100 11 12 FIGS.and Here, λ, λand λdenote each weight. After completion of training, in step, the computing devicemay input an input image to the trained deep learning-based Gaussian estimation model to generate a center probability map and a covariance map, and may post-process the center probability map and the covariance map to obtain 2D Gaussian parameters (a mean and a covariance).illustrate an example of visualizing 2D Gaussians. A 3D Gaussian may be generated by using such 2D Gaussians. This will be described in more detail below.

100 13 a FIG.() 13 b FIG.() First, the computing deviceextracts local maxima from the predicted center probability map.shows the predicted center probability map, and dots inindicate the local maxima.

100 13 c FIG.() 13 d FIG.() The computing devicesamples the predicted covariance map to extract a covariance at each local maximum. This is illustrated in. At this time, overlapping ellipses may exist, and unnecessary ellipses may be removed based on criteria such as overlap, size, and reliability. For example, when ellipses extracted at the local maxima substantially overlap one another, only the ellipse having the highest reliability (i.e., the highest center probability) may be left and the others may be removed. In addition, similarity between covariance matrices may be compared so that only one ellipse among similar ellipses is retained, an ellipse having a size that is too small may be regarded as noise and removed, and an ellipse having a size that is too large may be restricted because it may excessively represent a single object. Furthermore, when a probability value in the center probability map is less than a predetermined threshold, a corresponding ellipse may be removed, and singular values of a predicted covariance matrix may be analyzed so that an ellipse having low reliability is filtered out. A result of removing unnecessary ellipses from among the overlapping ellipses is shown in.

13 e FIG.() 13 f FIG.() By sampling colors of an RGB image at the local maxima that are not removed, a result as shown inmay be obtained, and the sampled colors are combined with the ellipses so that 2D Gaussians are defined (see).

13 f FIG.() 1 2 In, geometric parameters of each ellipse may include a center position (u, v), an orientation, and a major axis (l), and a minor axis (l).

−1 T 12 a FIG.() Lifting to a 3D space starts from back-projecting the center (u, v) into world-space coordinates by using a depth map D and a camera pose P provided by a tracking module. Defining a depth value d as D(u, v) and denoting a camera intrinsic matrix as K, dK(u, v, 1)represents a point in a camera coordinate system, which can be transformed into world-space coordinates by using P. In, (x, y, z) denotes world-space coordinates of the center.

14 b FIG.() 14 c FIG.() Using the depth map D, a normal map N is generated, and the ellipse in the world coordinate system is rotated so as to be aligned with the surface ga n (see). Next, as shown in, the major and minor axes are respectively scaled to

so that the size of the ellipse projected in the world coordinate system becomes as close as possible to that of the original 2D ellipse. To construct a 3D Gaussian, a third axis along the normal direction is required; this is set to

14 d FIG.() as illustrated in. Through this process, a 3D Gaussian is finally generated in the world coordinate system.

15 FIG. is a block diagram schematically illustrating an internal configuration of a computing device according to an embodiment of the present invention.

15 FIG. 100 1510 1520 1530 1540 1550 Referring to, a computing deviceaccording to an embodiment of the present invention includes a dataset configuration unit, a learning unit, a Gaussian estimation unit, a memory, and a processor.

1510 1 FIG. The dataset configuration unitrandomly generates a Voronoi diagram for an input image and configures a dataset by using the Voronoi diagram. Since this has already been described with reference to, a duplicate description will be omitted.

1520 1 FIG. The learning unitmay train a deep learning-based Gaussian estimation model by using the dataset so that the deep learning-based Gaussian estimation model outputs a center probability map and a covariance map for an input image. Since this has already been described with reference to, a duplicate description will be omitted.

1530 1 FIG. The Gaussian estimation unitinputs an input image to the trained deep learning-based Gaussian estimation model to generate a center probability map and a covariance map, and acquires 2D Gaussian parameters (a mean and a covariance) by post-processing the center probability map and the covariance map. This is also identical to what has been described with reference to, and thus a duplicate description will be omitted.

1540 The memorystores various instructions for performing the deep learning-based adaptive Gaussian estimation method according to an embodiment of the present invention.

1550 1510 1520 1530 1540 The processoris a means for controlling internal components according to an embodiment of the present invention (for example, the dataset configuration unit, the learning unit, the Gaussian estimation unit, the memory, etc.).

16 FIG. 16 FIG. is a view illustrating an example of extracting 2D Gaussians from general images according to an embodiment of the present invention. As shown in, it can be seen that 2D Gaussians can be extracted regardless of image style, such as persons, animals, structures (rooms), and cartoons.

The device and method according to the embodiments of the present disclosure may be implemented in a program that can be executed by various computers and may be recorded on computer-readable media. The computer-readable media may include program commands, data files, and data structures individually or in combinations thereof. The program commands that are recorded on a computer-readable media may be those specifically designed and configured for the present disclosure or may be those known to those engaged in the computer software field and thus available. The computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic media such as a magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program commands, such as ROM, RAM, and flash memory. The program commands include not only machine language codes compiled by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

The hardware device may be configured to operate as one or more software modules to perform the operation of the present disclosure, and vice versa.

The present disclosure was described above focusing on the embodiments thereof. It would be understood by those skilled in the art that the present disclosure may be implemented in a modified form without departing from the scope of the present disclosure. Therefore, the disclosed embodiments should be considered in terms of explaining, not limiting. The scope of the present disclosure is shown in the claims, not in the above description, and all differences within an equivalent range should be construed as being included in the present disclosure.