Technologies for improving the accuracy of depth camera images include a computing device to generate a foreground mask and a background mask for an image generated by a depth camera. The computing device identifies areas of a depth image of a depth channel of the generated image having unknown depth values as one of interior depth holes or exterior depth holes based on the foreground and background masks. The computing device fills at least a portion of the interior depth holes of the depth image based on depth values of areas of the depth image within a threshold distance of the corresponding portion of the interior depth holes. Similarly, the computing device fills at least a portion of the exterior depth holes of the depth image based on depth values of areas of the depth image within the threshold distance of the corresponding portion of the exterior depth holes.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A computing device for improving the accuracy of depth camera images, the computing device comprising: a processor; and a memory having stored therein: a mask generation module to (i) generate a foreground mask for an image generated by a depth camera, wherein the generated image includes a depth channel and the foreground mask includes at least one foreground object of the generated image, (ii) generate a background mask for the generated image, wherein the background mask includes portions of the generated image other than the at least one foreground object, (iii) generate a binary mask including a first plurality of pixels, each pixel of the first plurality of pixels having a corresponding known depth value, (iv) expand the generated binary mask, (v), intersect the expanded binary mask with a second plurality of pixels to generate a reduced plurality of pixels, each pixel of the second plurality of pixels and the reduced plurality of pixels having an unknown depth value, (vi) apply the foreground mask to the reduced plurality of pixels to identify interior depth holes, and (vii) apply the background mask to the reduced plurality of pixels to identify exterior depth holes; and an image filling module to (i) fill at least a portion of the interior depth holes of the depth image based on depth values of areas of the depth image within a threshold distance of the corresponding portion of the interior depth holes and (ii) fill at least a portion of the exterior depth holes of the depth image based on depth values of areas of the depth image within the threshold distance of the corresponding portion of the exterior depth holes.
A computing device enhances depth camera image accuracy by identifying and filling depth holes. It generates a foreground mask (identifying objects) and a background mask. A binary mask represents known depth values. This mask is expanded and then intersected with unknown depth pixels, creating a set of pixels with missing depth. The foreground/background masks classify these missing pixels as either "interior" or "exterior" depth holes. The system fills these holes by using nearby known depth values within a threshold distance. Interior holes are filled using only surrounding interior depth values, and exterior holes are filled using only surrounding exterior depth values to refine depth estimations.
2. The computing device of claim 1 , wherein to generate the foreground mask comprises to: identify edges of the at least one foreground object; and generate a second binary mask of the at least one object based on the identified edges.
To create the foreground mask (from the device improving depth camera accuracy), the system identifies the edges of foreground objects and generates a binary mask specifically highlighting these objects based on those detected edges. This focused mask aids in distinguishing foreground elements when classifying depth holes.
3. The computing device of claim 2 , wherein to identify the edges of the at least one foreground object comprises to apply an edge detection filter to the generated image.
This computing device improves the accuracy of depth camera images. It includes a mask generation module that processes images from a depth camera, which contain a depth channel. The module generates a foreground mask highlighting at least one foreground object in the image. This foreground mask generation involves identifying the edges of the foreground object. Specifically, to identify these edges, the device applies an edge detection filter to the original depth camera image. After edges are identified, a second binary mask of the foreground object is created based on these edges. The device also generates a background mask and uses these masks to identify and fill interior and exterior depth holes in the image.
4. The computing device of claim 3 , wherein to apply the edge detection filter comprises to apply a separable Sobel filter to the generated image.
In the edge detection process (of the system improving depth camera accuracy), a separable Sobel filter is applied to the depth image. This specific filter efficiently detects edges in both horizontal and vertical directions, improving the accuracy of identified foreground object boundaries.
5. The computing device of claim 3 , wherein to generate the second binary mask comprises to: connect the identified edges of the at least one foreground object to identify boundaries of the at least one foreground object; and flood-fill the at least one bounded foreground object.
To generate the foreground object binary mask (in the system improving depth camera accuracy), the system connects the identified edges to form complete object boundaries and then flood-fills the area enclosed by these boundaries. This ensures that the entire foreground object is consistently represented in the mask.
6. The computing device of claim 5 , wherein to identify the boundaries of the at least one foreground object comprises to: convolve the edge detection filter with the generated image; and filter out low-valued pixels of a corresponding image response.
In the process of identifying foreground object boundaries (for improved depth camera accuracy), the edge detection filter is convolved with the depth image. Low-valued pixels in the resulting image response are then filtered out, refining the boundary detection by eliminating noise and less significant edge features.
7. The computing device of claim 1 , wherein to expand the generated binary mask comprises to perform a dilation operation.
The binary mask, representing known depth values, is expanded (in the system improving depth camera accuracy) using a dilation operation. This operation increases the size of the known regions, facilitating the identification of nearby unknown depth pixels for subsequent filling.
8. The computing device of claim 1 , wherein to fill at least the portion of the interior depth holes and to fill at least the portion of the exterior depth holes comprises to apply a weighting function to a neighborhood of each of the interior depth holes and each of the exterior depth holes, the weighting function having a convolution kernel size defining the threshold and the neighborhood.
The process of filling interior and exterior depth holes (in the system improving depth camera accuracy) involves applying a weighting function to the surrounding neighborhood of each hole. The size of the convolution kernel used in the weighting function defines the threshold distance and neighborhood considered for depth value interpolation, ensuring accurate hole filling based on local context.
9. The computing device of claim 8 , wherein to fill at least the portion of the interior depth holes comprises to apply the weighting function to the neighborhood of each of the interior depth holes, such that the weighting function ignores (i) unknown depth values in the neighborhood and (ii) exterior depth holes; and wherein to fill at least the portion of the exterior depth holes comprises to apply the weighting function to the neighborhood of each of the exterior depth holes, such that the weighting function ignores (i) unknown depth values in the neighborhood and (ii) interior depth holes.
When filling interior depth holes (in the system improving depth camera accuracy), the weighting function only considers known depth values within the neighborhood and ignores both unknown depth values and exterior depth holes. Conversely, when filling exterior depth holes, the weighting function ignores unknown depth values and interior depth holes. This selective approach ensures that hole filling is based on appropriate contextual data for each hole type.
10. The computing device of claim 9 , wherein to fill at least the portion of the interior depth holes and to fill at least the portion of the exterior depth holes comprises to leave unfilled a depth hole having a smaller weighting function value than a threshold number.
To refine the hole-filling process (in the system improving depth camera accuracy), a depth hole is left unfilled if the weighting function produces a value lower than a specified threshold. This prevents inaccurate or unreliable depth estimations from being introduced into the depth image, maintaining data quality.
11. One or more non-transitory machine readable storage media comprising a plurality of instructions stored thereon that, in response to execution by a computing device, cause the computing device to: generate a foreground mask for an image generated by a depth camera, the generated image including a depth channel and the foreground mask including at least one foreground object of the generated image; generate a background mask for the generated image, the background mask including portions of the generated image other than the at least one foreground object; generate a binary mask including a first plurality of pixels, each pixel of the first plurality of pixels having a corresponding known depth value; expand the generated binary mask; intersect the expanded binary mask with a second plurality of pixels to generate a reduced plurality of pixels, each pixel of the second plurality of pixels and the reduced plurality of pixels having an unknown depth value; apply the foreground mask to the reduced plurality of pixels to identify interior depth holes; apply the background mask to the reduced plurality of pixels to identify exterior depth holes; fill at least a portion of the interior depth holes of the depth image based on depth values of areas of the depth image within a threshold distance of the corresponding portion of the interior depth holes; and fill at least a portion of the exterior depth holes of the depth image based on depth values of areas of the depth image within the threshold distance of the corresponding portion of the exterior depth holes.
A non-transitory computer-readable medium stores instructions that, when executed, enhance depth camera image accuracy. The process involves generating a foreground mask (identifying objects) and a background mask. A binary mask represents known depth values and is expanded and then intersected with unknown depth pixels, creating a set of pixels with missing depth. The foreground/background masks classify these missing pixels as either "interior" or "exterior" depth holes. The system fills these holes by using nearby known depth values within a threshold distance. Interior holes are filled using only surrounding interior depth values, and exterior holes are filled using only surrounding exterior depth values to refine depth estimations.
12. The one or more non-transitory machine readable storage media of claim 11 , wherein to generate the foreground mask comprises to: identify edges of the at least one foreground object; and generate a second binary mask of the at least one object based on the identified edges.
To create the foreground mask (from the process improving depth camera accuracy), the system identifies the edges of foreground objects and generates a binary mask specifically highlighting these objects based on those detected edges. This focused mask aids in distinguishing foreground elements when classifying depth holes.
13. The one or more non-transitory machine readable storage media of claim 12 , wherein identifying the edges of the at least one foreground object comprises applying an edge detection filter to the generated image.
The system that improves depth camera accuracy identifies edges of foreground objects for foreground mask generation by applying an edge detection filter to the depth image. This filter highlights rapid changes in pixel intensity, indicating object boundaries within the image data.
14. The one or more non-transitory machine readable storage media of claim 13 , wherein to generate the second binary mask comprises to: connect the identified edges of the at least one foreground object to identify boundaries of the at least one foreground object; and flood-fill the at least one bounded foreground object.
To generate the foreground object binary mask (in the system improving depth camera accuracy), the system connects the identified edges to form complete object boundaries and then flood-fills the area enclosed by these boundaries. This ensures that the entire foreground object is consistently represented in the mask.
15. The one or more non-transitory machine readable storage media of claim 14 , wherein to identify the boundaries of the at least one foreground object comprises to: convolve the edge detection filter with the generated image; and filter out low-valued pixels of a corresponding image response.
In the process of identifying foreground object boundaries (for improved depth camera accuracy), the edge detection filter is convolved with the depth image. Low-valued pixels in the resulting image response are then filtered out, refining the boundary detection by eliminating noise and less significant edge features.
16. The one or more non-transitory machine readable storage media of claim 11 , wherein to expand the generated binary mask comprises to perform a dilation operation.
The binary mask, representing known depth values, is expanded (in the system improving depth camera accuracy) using a dilation operation. This operation increases the size of the known regions, facilitating the identification of nearby unknown depth pixels for subsequent filling.
17. The one or more non-transitory machine readable storage media of claim 11 , wherein to fill at least the portion of the interior depth holes and to fill at least the portion of the exterior depth holes comprises to apply a weighting function to a neighborhood of each of the interior depth holes and each of the exterior depth holes, the weighting function having a convolution kernel size defining the threshold and the neighborhood.
The process of filling interior and exterior depth holes (in the system improving depth camera accuracy) involves applying a weighting function to the surrounding neighborhood of each hole. The size of the convolution kernel used in the weighting function defines the threshold distance and neighborhood considered for depth value interpolation, ensuring accurate hole filling based on local context.
18. The one or more non-transitory machine readable storage media of claim 17 , wherein to fill at least the portion of the interior depth holes comprises to apply the weighting function to the neighborhood of each of the interior depth holes, such that the weighting function ignores (i) unknown depth values in the neighborhood and (ii) exterior depth holes; and wherein to fill at least the portion of the exterior depth holes comprises to apply the weighting function to the neighborhood of each of the exterior depth holes, such that the weighting function ignores (i) unknown depth values in the neighborhood and (ii) interior depth holes.
When filling interior depth holes (in the system improving depth camera accuracy), the weighting function only considers known depth values within the neighborhood and ignores both unknown depth values and exterior depth holes. Conversely, when filling exterior depth holes, the weighting function ignores unknown depth values and interior depth holes. This selective approach ensures that hole filling is based on appropriate contextual data for each hole type.
19. The one or more non-transitory machine readable storage media of claim 18 , wherein to fill at least the portion of the interior depth holes and to fill at least the portion of the exterior depth holes comprises to leave unfilled a depth hole having a smaller weighting function value than a threshold number.
To refine the hole-filling process (in the system improving depth camera accuracy), a depth hole is left unfilled if the weighting function produces a value lower than a specified threshold. This prevents inaccurate or unreliable depth estimations from being introduced into the depth image, maintaining data quality.
20. A method for improving the accuracy of depth camera images on a computing device, the method comprising: generating, on the computing device, a foreground mask for an image generated by a depth camera, the generated image including a depth channel and the foreground mask including at least one foreground object of the generated image; generating, on the computing device, a background mask for the generated image, the background mask including portions of the generated image other than the at least one foreground object; generating, on the computing device, a binary mask including a first plurality of pixels, each pixel of the first plurality of pixels having a corresponding known depth value; expanding, on the computing device, the generated binary mask; intersecting, on the computing device, the expanded binary mask with a second plurality of pixels to generate a reduced plurality of pixels, each pixel of the second plurality of pixels and the reduced plurality of pixels having an unknown depth value; applying, on the computing device, the foreground mask to the reduced plurality of pixels to identify interior depth holes; applying, on the computing device, the background mask to the reduced plurality of pixels to identify exterior depth holes; filling, on the computing device, at least a portion of the interior depth holes of the depth image based on depth values of areas of the depth image within a threshold distance of the corresponding portion of the interior depth holes; and filling, on the computing device, at least a portion of the exterior depth holes of the depth image based on depth values of areas of the depth image within the threshold distance of the corresponding portion of the exterior depth holes.
A method enhances depth camera image accuracy. It generates a foreground mask (identifying objects) and a background mask. A binary mask represents known depth values. This mask is expanded and then intersected with unknown depth pixels, creating a set of pixels with missing depth. The foreground/background masks classify these missing pixels as either "interior" or "exterior" depth holes. The method fills these holes by using nearby known depth values within a threshold distance. Interior holes are filled using only surrounding interior depth values, and exterior holes are filled using only surrounding exterior depth values to refine depth estimations.
21. The method of claim 20 , wherein generating the foreground mask comprises: identifying edges of the at least one foreground object by applying an edge detection filter to the generated image; and generating a second binary mask of the at least one object based on the identified edges by (i) connecting the identified edges of the at least one foreground object to identify boundaries of the at least one foreground object and (ii) flood-filling the at least one bounded foreground object.
Foreground mask generation for improving depth camera accuracy involves two steps: (1) Identifying the edges of foreground objects by applying an edge detection filter to the image and (2) Generating a binary mask of the object based on the identified edges. This includes connecting the identified edges to identify the object's boundaries, and then flood-filling the bounded area to create a complete mask.
22. The method of claim 20 , wherein filling at least the portion of the interior depth holes and filling at least the portion of the exterior depth holes comprises applying a weighting function to a neighborhood of each of the interior depth holes and each of the exterior depth holes, the weighting function having a convolution kernel size defining the threshold and the neighborhood.
Filling interior and exterior depth holes to improve depth camera accuracy involves applying a weighting function to a neighborhood around each hole. The weighting function's convolution kernel size defines both the threshold distance and the neighborhood size used in the calculation, ensuring appropriate consideration of nearby pixels for accurate depth estimation.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
May 20, 2013
March 21, 2017
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.