Modifying Two-Dimensional Images Utilizing Iterative Three-Dimensional Meshes of the Two-Dimensional Images

1. A system comprising: a memory component; and a processing device coupled to the memory component, the processing device to perform operations comprising: determining, utilizing one or more neural networks, first pixel depth values representing first estimated depths at pixels of a two-dimensional image; generating, utilizing the one or more neural networks, a first three-dimensional mesh by generating a first tessellation incorporating depth information from the first pixel depth values of the two-dimensional image; determining, in response to a displacement input to a portion of the two-dimensional image within a graphical user interface displaying the two-dimensional image, that the two-dimensional image is modified to create a modified two-dimensional image comprising at least one modified portion of the two-dimensional image according to a corresponding modified portion of the first three-dimensional mesh; determining, utilizing the one or more neural networks, second pixel depth values representing second estimated depths at pixels of the modified two-dimensional image; and generating, utilizing the one or more neural networks in response to determining that the two-dimensional image is modified to create the modified two-dimensional image, a second three-dimensional mesh by generating a second tessellation incorporating depth information of the modified two-dimensional image based on the second pixel depth values for the modified two-dimensional image according to the at least one modified portion of the modified two-dimensional image.

2. The system of claim 1, wherein: generating the first three-dimensional mesh comprises generating the first tessellation based on a first set of sampled points of the two-dimensional image; and generating the second three-dimensional mesh comprises generating the second tessellation based on a second set of sampled points of the modified two-dimensional image.

3. The system of claim 2, wherein: generating the first three-dimensional mesh comprises determining, utilizing the one or more neural networks, the first pixel depth values according to estimated camera parameters corresponding to a viewpoint of the two-dimensional image; and generating the second three-dimensional mesh comprises determining, utilizing the one or more neural networks, the second pixel depth values according to the estimated camera parameters corresponding to the viewpoint of the two-dimensional image.

4. The system of claim 1, wherein generating the second three-dimensional mesh comprises generating the second three-dimensional mesh in response to a request to commit the at least one modified portion of the two-dimensional image.

5. The system of claim 1, wherein determining the modified two-dimensional image comprises: determining a two-dimensional position of the displacement input within the two-dimensional image; determining a three-dimensional position of the displacement input on the first three-dimensional mesh based on the two-dimensional position of the displacement input within the two-dimensional image; and determining the at least one modified portion of the two-dimensional image at the two-dimensional position based on a modified portion of the first three-dimensional mesh at the three-dimensional position according to the displacement input.

6. The system of claim 5, wherein determining the modified two-dimensional image comprises: determining a displacement direction of the displacement input relative to the first three-dimensional mesh; and determining the modified portion of the first three-dimensional mesh according to the displacement direction of the displacement input.

7. The system of claim 1, wherein generating the second three-dimensional mesh comprises interpolating, in connection with the at least one modified portion of the two-dimensional image, vertex positions of a plurality of vertices in a portion of the second three-dimensional mesh corresponding to an obscured portion of the two-dimensional image.

8. The system of claim 1, wherein determining the modified two-dimensional image comprises generating, utilizing an inpainting neural network, inpainted image content for the at least one modified portion of the two-dimensional image in response to detecting an artifact in the at least one modified portion of the two-dimensional image.

9. A non-transitory computer readable medium comprising instructions, which when executed by a processing device, cause the processing device to perform operations comprising: determining, utilizing one or more neural networks, pixel depth values representing estimated depths at pixels of a two-dimensional image; generating, utilizing the one or more neural networks, a three-dimensional mesh by generating a tessellation incorporating depth information from the pixel depth values of the two-dimensional image; determining that the two-dimensional image is modified to create a modified two-dimensional image comprising at least one modified portion of the two-dimensional image in response to a displacement input to a portion of the two-dimensional image within a graphical user interface displaying the two-dimensional image; determining, utilizing the one or more neural networks, new pixel depth values representing new estimated depths at pixels of the modified two-dimensional image; and generating, utilizing the one or more neural networks in response to determining that the two-dimensional image is modified to create the modified two-dimensional image, an updated three-dimensional mesh by generating an updated tessellation incorporating depth information of the modified two-dimensional image based on the new pixel depth values for the modified two-dimensional image according to the at least one modified portion of the modified two-dimensional image.

10. The non-transitory computer readable medium of claim 9, wherein generating the three-dimensional mesh comprises: generating the three-dimensional mesh comprises: sampling a first set of points of the two-dimensional image according to first density values corresponding to pixels of the two-dimensional image; and generating the three-dimensional mesh based on the first set of points sampled in the two-dimensional image; and generating the updated three-dimensional mesh comprises: sampling a second set of points of the modified two-dimensional image according to second density values corresponding to pixels of the modified two-dimensional image; and generating the updated three-dimensional mesh based on the second set of points sampled in the modified two-dimensional image.

11. The non-transitory computer readable medium of claim 9, wherein determining the modified two-dimensional image comprises: determining a displaced portion of the three-dimensional mesh based on one or more displacement directions of the displacement input; and determining the at least one modified portion of the two-dimensional image based on the displaced portion of the three-dimensional mesh.

12. The non-transitory computer readable medium of claim 9, wherein generating the updated three-dimensional mesh comprises determining the new pixel depth values for the modified two-dimensional image in response to detecting an action to commit the at least one modified portion.

13. A method comprising: determining, by at least one processor utilizing one or more neural networks, pixel depth values representing estimated depths at pixels of a two-dimensional image; generating, by the at least one processor utilizing the one or more neural networks, a three-dimensional mesh by generating a tessellation incorporating depth information from the pixel depth values of the two-dimensional image; determining, by the at least one processor, that the two-dimensional image is modified to create a modified two-dimensional image comprising at least one modified portion of the two-dimensional image in response to a displacement input to a portion of the two-dimensional image within a graphical user interface displaying the two-dimensional image; determining, by the at least one processor utilizing the one or more neural networks, new pixel depth values representing new estimated depths at pixels of the modified two-dimensional image; and generating, by the at least one processor utilizing the one or more neural networks in response to determining that the two-dimensional image is modified to create the modified two-dimensional image, an updated three-dimensional mesh by generating an updated tessellation incorporating depth information of the modified two-dimensional image based on the new pixel depth values for the modified two-dimensional image according to the at least one modified portion of the modified two-dimensional image.

14. The method of claim 13, wherein generating the three-dimensional mesh comprises: generating the three-dimensional mesh by determining displacement of vertices of the tessellation from the depth information of the two-dimensional image based on the pixel depth values and estimated camera parameters; or generating the three-dimensional mesh based on a plurality of points sampled in the two-dimensional image according to density values determined from the pixel depth values of the two-dimensional image.

15. The method of claim 13, wherein determining the modified two-dimensional image comprises: determining a displaced portion of the three-dimensional mesh based on the displacement input to the portion of the two-dimensional image; and generating the modified two-dimensional image according to the displaced portion of the three-dimensional mesh.

16. The method of claim 15, wherein generating the updated three-dimensional mesh comprises: sampling a plurality of points of the modified two-dimensional image according to density values determined from the new pixel depth values corresponding to the pixels of the modified two-dimensional image; and generating the updated three-dimensional mesh based on the plurality of points sampled in the modified two-dimensional image.

17. The method of claim 13, wherein generating the updated three-dimensional mesh comprises: detecting an action to generate the modified two-dimensional image by committing a displacement of the at least one modified portion to the two-dimensional image; and generating the updated three-dimensional mesh in response to committing the displacement of the at least one modified portion to the two-dimensional image.

18. The method of claim 13, wherein generating the updated three-dimensional mesh comprises: determining that an initial position of the at least one modified portion of the two-dimensional image obscures an additional portion of the two-dimensional image; and generating the updated three-dimensional mesh by interpolating vertex positions in a portion of the three-dimensional mesh corresponding to the additional portion of the two-dimensional image obscured by the initial position of the at least one modified portion of the two-dimensional image.

19. The method of claim 13, wherein generating the modified two-dimensional image comprises: determining that the at least one modified portion of the two-dimensional image comprises an image artifact; and generating, utilizing an inpainting neural network, inpainted image content correcting the image artifact within the at least one modified portion.

20. The method of claim 13, further comprising generating, in a plurality of displacement iterations comprising a plurality of displacement inputs within the graphical user interface, a plurality of updated three-dimensional meshes corresponding to a plurality of modified two-dimensional images in connection with the modified two-dimensional image.