Depth Rate Up-Conversion

1. A method comprising: at an electronic device having a processor: receiving a first sequence of images corresponding to light intensities of a physical environment and a second sequence of images corresponding to depths of the physical environment, wherein the first sequence has a first rate and the second sequence has a second rate that differs from the first rate; receiving mask data distinguishing a portion of the physical environment from a portion of virtual content; generating a contour image based on the mask data for a first frame of the first sequence of images, the contour image identifying an interior edge or exterior edge of the portion of the physical environment or the portion of the virtual content; adjusting the contour image based on a second frame of the first sequence of images; and determining additional depth images corresponding to depths of the physical environment based on the adjusting of the contour image.

2. The method of claim 1 , further comprising adjusting the second sequence of images based at least on the additional depth images.

3. The method of claim 1 , further comprising upconverting the second rate of the second sequence of images by inserting the additional depth images into the second sequence of images, wherein the upconverted second rate is synchronized with the first rate.

4. The method of claim 1 , further comprising: adjusting the second sequence of images based at least on the additional depth images; receiving a third sequence of images corresponding to depths of the virtual content; and determining occlusion between the physical environment and the virtual content based on the adjusted second sequence of images and the third sequence of images.

5. The method of claim 4 , wherein determining the occlusion comprises comparing depths of the physical environment in the adjusted second sequence of images with depths of the virtual content in the third sequence of images.

6. The method of claim 1 , further comprising receiving a third sequence of images corresponding to light intensities of the virtual content and a fourth sequence of images corresponding to depths of the virtual content.

7. The method of claim 6 , wherein the mask data comprises an occlusion mask that identifies an area of the virtual content depicted in an image of the third sequence of images that is occluded by an area of the physical environment depicted in a corresponding image of the first sequence of images.

8. The method of claim 7 , wherein generating a contour image comprises identifying an inside outline inside the area of the virtual content that is occluded by the area of the physical environment in the occlusion mask.

9. The method of claim 7 , wherein generating a contour image comprises identifying an outside outline outside the area of the virtual content that is occluded by the area of the physical environment in the occlusion mask.

10. The method of claim 1 , wherein: the contour image for the first frame of the first sequence of images comprises a first contour image; adjusting the contour image based on the second frame of the first sequence of images comprises a second contour image; the second frame of the first sequence of images subsequently follows the first frame; and determining the additional depth images comprises: comparing a plurality of pixels of the first contour image of the first frame with a plurality of pixels of the contour image of the second frame for the first sequence of images, wherein the plurality of pixels of the first contour image are spatially correlated and are aligned with the plurality of pixels of the second contour image; identifying a movement of a feature of the physical environment from a first pixel in the first contour image to an associated second pixel in the second contour image; and generating, based on the identified movement of the feature, the additional depth images.

11. The method of claim 1 , wherein pixels of the images of the first sequence are spatially correlated and aligned with pixels of images of the second sequence.

12. The method of claim 1 , wherein the first frame of the first sequence of images is synchronized with a first frame of the second sequence of images.

13. The method of claim 1 , further comprising providing a computer-generated reality (CGR) environment that includes the virtual content and the physical environment.

14. The method of claim 12 , wherein the CGR environment is provided on a head-mounted device (HMD).

15. A device comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising: receiving a first sequence of images corresponding to light intensities of a physical environment and a second sequence of images corresponding to depths of the physical environment, wherein the first sequence has a first rate and the second sequence has a second rate that differs from the first rate; receiving mask data distinguishing a portion of the physical environment from a portion of virtual content; generating a contour image based on the mask data for a first frame of the first sequence of images, the contour image identifying an interior edge or exterior edge of the portion of the physical environment or the portion of the virtual content; adjusting the contour image based on a second frame of the first sequence of images; and determining additional depth images corresponding to depths of the physical environment based on the adjusting of the contour image.

16. The device of claim 15 , the operations further comprising: adjusting the second sequence of images based at least on the additional depth images; and upconverting the second rate of the second sequence of images by inserting the additional depth images into the second sequence of images, wherein the upconverted second rate is synchronized with the first rate.

17. The device of claim 15 , the operations further comprising receiving a third sequence of images corresponding to light intensities of the virtual content and a fourth sequence of images corresponding to depths of the virtual content.

18. The device of claim 17 , wherein the mask data comprises an occlusion mask that identifies an area of the virtual content depicted in an image of the third sequence of images that is occluded by an area of the physical environment depicted in a corresponding image of the first sequence of images.

19. The device of claim 18 , wherein: the contour image for the first frame of the first sequence of images comprises a first contour image; adjusting the contour image based on the second frame of the first sequence of images comprises a second contour image; the second frame of the first sequence of images subsequently follows the first frame; and determining the additional depth images based on the adjusting of the contour image comprises: comparing a plurality of pixels of the first contour image of the first frame with a plurality of pixels of the contour image of the second frame for the first sequence of images, wherein the plurality of pixels of the first contour image are spatially correlated and are aligned with the plurality of pixels of the second contour image; identifying a movement of a feature of the physical environment from a first pixel in the first contour image to an associated second pixel in the second contour image; and generating, based on the identified movement of the feature, the additional depth images.

20. A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising: receiving a first sequence of images corresponding to light intensities of a physical environment and a second sequence of images corresponding to depths of the physical environment, wherein the first sequence has a first rate and the second sequence has a second rate that differs from the first rate; receiving mask data distinguishing a portion of the physical environment from a portion of virtual content; generating a contour image based on the mask data for a first frame of the first sequence of images, the contour image identifying an interior edge or exterior edge of the portion of the physical environment or the portion of the virtual content; adjusting the contour image based on a second frame of the first sequence of images; and determining additional depth images corresponding to depths of the physical environment based on the adjusting of the contour image.