A 3D conversation system can facilitate 3D conversations in an augmented reality environment, allowing conversation participants to appear as if they are face-to-face. The 3D conversation system can accomplish this with a pipeline of data processing stages, which can include calibrate, capture, tag and filter, compress, decompress, reconstruct, render, and display stages. Generally, the pipeline can capture images of the sending user, create intermediate representations, transform the representations to convert from the orientation the images were taken from to a viewpoint of the receiving user, and output images of the sending user, from the viewpoint of the receiving user, in synchronization with audio captured from the sending user. Such a 3D conversation can take place between two or more sender/receiving systems and, in some implementations can be mediated by one or more server systems. In various configurations, stages of the pipeline can be customized based on a conversation context.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
2. The computer-readable storage medium of claim 1, wherein the reconstruction system transforms the version of the transmission data into the 3D representation in part by both X) combining data, captured by multiple sources, from the version of the transmission data and Y) determining position and contour information in 3D space for aspects of the transmission data.
This invention relates to a system for reconstructing three-dimensional (3D) representations from transmission data, addressing challenges in accurately capturing and processing spatial information from multiple data sources. The system transforms transmission data into a 3D representation by combining data captured by multiple sources and determining position and contour information in 3D space for aspects of the transmission data. The reconstruction process involves integrating data from diverse sources to enhance accuracy and detail in the 3D output. Position and contour information are derived to define the spatial relationships and boundaries of objects within the 3D space, ensuring precise reconstruction. The system may be used in applications such as medical imaging, industrial inspection, or virtual reality, where accurate 3D modeling from multiple data inputs is critical. The invention improves upon prior methods by leveraging multi-source data fusion and advanced spatial analysis to produce high-fidelity 3D representations.
4. The computer-readable storage medium of claim 1, wherein the reconstruction system applies shading and/or color data onto the 3D representation, using the calibration data to map portions of the color information to the 3D representation.
This invention relates to a system for reconstructing three-dimensional (3D) representations of objects from captured data, particularly focusing on enhancing the visual accuracy of the reconstructed models. The system addresses the challenge of accurately mapping color and shading information onto 3D representations derived from captured data, such as images or depth scans, to produce realistic and visually coherent models. The system includes a reconstruction module that processes captured data to generate a 3D representation of an object. A calibration module generates calibration data that defines the spatial relationship between the captured data and the 3D representation. This calibration data is used to align and map color and shading information from the captured data onto the 3D representation. The shading and color data may be derived from multiple sources, such as images or other sensor inputs, and the calibration data ensures that the color and shading are accurately positioned on the 3D model, avoiding misalignment or distortion. The system may also include a rendering module that applies the mapped color and shading data to the 3D representation, producing a final output that is visually accurate and consistent. The calibration data may be generated using techniques such as feature matching, geometric alignment, or sensor fusion, depending on the type of captured data and the desired level of precision. The invention improves the realism and usability of 3D reconstructions by ensuring that visual attributes are correctly mapped to the 3D structure.
5. The computer-readable storage medium of claim 1, wherein the 3D conversation data was acquired from multiple depth sensors and wherein the process further comprises combining the data from the multiple depth sensors into the depth information.
This invention relates to processing three-dimensional (3D) conversation data acquired from multiple depth sensors to generate depth information for applications such as virtual reality, augmented reality, or human-computer interaction. The problem addressed is the challenge of accurately capturing and integrating 3D data from multiple depth sensors to create a coherent representation of a scene or interaction. The invention involves a computer-readable storage medium containing instructions for processing 3D conversation data. The data is collected from multiple depth sensors, which may include devices like depth cameras or LiDAR systems. The process includes combining the data from these sensors to generate depth information, which represents the spatial relationships and distances of objects or participants in the scene. This combined depth information can be used to enhance virtual environments, improve gesture recognition, or enable more immersive human-computer interactions. The system ensures that data from different sensors is synchronized and merged accurately, accounting for variations in sensor positions, orientations, and calibration. This allows for a more precise and reliable 3D reconstruction of the environment or interaction. The combined depth information can then be used in applications requiring spatial awareness, such as virtual avatars, collaborative virtual spaces, or real-time motion tracking. The invention improves the accuracy and robustness of 3D data processing in dynamic environments where multiple sensors are deployed.
6. The computer-readable storage medium of claim 5, wherein the combining the data from the multiple depth sensors comprises compensating for distances between the multiple depth sensors by applying, based on the calibration data, a transformation to the data from one or more of the multiple depth sensors.
This invention relates to a system for processing depth sensor data in a multi-sensor environment. The problem addressed is the challenge of accurately combining depth data from multiple sensors when they are positioned at different locations, leading to misalignment or distortion in the combined output. The solution involves a computer-readable storage medium containing instructions for a processor to perform operations that include calibrating the multiple depth sensors to generate calibration data, then combining the data from these sensors while compensating for their spatial separation. The compensation step applies a transformation to the data from one or more sensors based on the calibration data, ensuring that the combined depth information is spatially consistent. The calibration data may include positional and orientation information of each sensor, allowing precise alignment of the depth maps. This approach enables accurate 3D reconstruction or scene analysis in applications where multiple depth sensors are used, such as robotics, augmented reality, or medical imaging. The transformation may involve geometric adjustments like translation, rotation, or scaling to correct for sensor misalignment. The system ensures that the combined depth data reflects the true spatial relationships between objects in the environment.
9. The computer-readable storage medium of claim 1, wherein the 3D representation is one or more of: a point cloud, a signed distance function, populated voxels, a mesh, a light field or any combination thereof.
This invention relates to computer graphics and 3D data representation, specifically addressing the challenge of efficiently storing and processing diverse 3D data formats. The technology provides a computer-readable storage medium containing instructions for generating and manipulating 3D representations, which can take multiple forms. These representations include point clouds, signed distance functions, populated voxels, meshes, light fields, or any combination thereof. The system allows for flexible conversion between these formats, enabling compatibility with different rendering and processing pipelines. By supporting various 3D data structures, the invention facilitates applications in virtual reality, augmented reality, 3D modeling, and simulation, where different formats may be preferred for specific tasks. The storage medium also includes instructions for optimizing the representation based on factors such as memory usage, rendering speed, or data fidelity, ensuring efficient handling of complex 3D scenes. This adaptability makes the system suitable for real-time applications requiring dynamic switching between formats. The invention improves upon prior art by unifying multiple 3D representation techniques into a single, versatile framework, reducing the need for separate tools or conversions.
12. The computer-readable storage medium of claim 1, wherein the calibration data comprises one or more of: measurements of camera components, one or more captured images of a pre-defined target, heat data, moisture data, geographic mapping data for the one or more cameras, time-of-flight measurements, lighting conditions, or any combination thereof.
This invention relates to a computer-readable storage medium containing calibration data for camera systems, addressing the challenge of ensuring accurate and reliable camera performance across varying environmental and operational conditions. The calibration data includes measurements of camera components, such as lens distortion, sensor sensitivity, and alignment, to correct inherent hardware imperfections. It also incorporates captured images of predefined targets, which are used to validate and refine camera accuracy. Environmental factors like heat, moisture, and lighting conditions are recorded to account for their impact on image quality. Geographic mapping data and time-of-flight measurements help determine camera positioning and depth perception. By integrating these diverse data types, the system enables dynamic adjustments to camera settings, improving image clarity, precision, and consistency. This approach is particularly useful in applications requiring high accuracy, such as surveillance, autonomous vehicles, and industrial automation, where environmental variability can degrade performance. The invention ensures cameras maintain optimal functionality by continuously updating calibration parameters based on real-world conditions.
15. The method of claim 13, wherein the output of the one or more 2D images includes a wearable projection system projecting light, based on the one or more 2D images, into at least one eye of a user of the recipient system.
This invention relates to a wearable projection system for displaying visual information to a user. The system addresses the challenge of providing immersive, high-quality visual experiences in a compact, wearable form factor. The method involves generating one or more 2D images and projecting light based on these images directly into at least one eye of the user. The projection system is part of a recipient system that processes input data, such as sensor data or digital content, to produce the 2D images. The system may include optical components to direct the projected light into the user's eye, ensuring proper alignment and focus. The wearable design allows for hands-free operation, making it suitable for applications like augmented reality, virtual reality, or medical visualization. The projection system may incorporate adaptive optics to correct for individual user differences, such as eye shape or refractive errors, to enhance image clarity. The method ensures that the projected light is synchronized with the user's gaze or head movements, providing a seamless and immersive visual experience. The system may also include feedback mechanisms to adjust projection parameters in real time, improving comfort and performance. This approach enables lightweight, high-resolution displays that can be integrated into glasses, helmets, or other wearable devices.
16. The method of claim 13, wherein the transforming the version of the transmission data into the 3D representation is performed by X) combining data, captured by multiple sources, from the version of the transmission data and/or Y) determining position and contour information in 3D space for aspects of the transmission data.
This invention relates to generating three-dimensional (3D) representations from transmission data, such as medical imaging data. The problem addressed is the need to accurately reconstruct 3D models from data captured by multiple sources, such as sensors or imaging devices, to improve visualization and analysis. The method involves transforming transmission data into a 3D representation through two key processes. First, data from multiple sources is combined to enhance the accuracy and completeness of the 3D model. This may involve aligning, merging, or interpolating data from different perspectives or modalities. Second, the method determines position and contour information in 3D space for aspects of the transmission data, allowing for precise mapping of structures or features within the data. This step may involve identifying key points, surfaces, or boundaries in the data to define the 3D shape. The combined data and positional information are used to generate a detailed 3D representation, which can be used for applications such as medical diagnostics, surgical planning, or industrial inspections. The approach ensures that the 3D model accurately reflects the original transmission data, improving reliability and usability.
18. The computing system of claim 17, wherein the transforming the version of the transmission data into the 3D representation is performed by X) combining data, captured by multiple sources, from the version of the transmission data and/or Y) determining position and contour information in 3D space for aspects of the transmission data.
This invention relates to computing systems for processing transmission data, particularly for generating 3D representations from such data. The problem addressed is the need to accurately reconstruct and visualize complex transmission data, such as signals or sensor inputs, in three-dimensional space for applications like medical imaging, environmental monitoring, or industrial inspections. The system processes transmission data, which may originate from multiple sources, and transforms it into a 3D representation. This transformation involves either combining data from multiple sources to enhance accuracy or determining precise position and contour information in 3D space for key aspects of the transmission data. The combined approach ensures that the 3D representation is both spatially accurate and detailed, capturing the full complexity of the original transmission data. The system may also include preprocessing steps to filter or normalize the transmission data before transformation, ensuring consistency and reducing noise. Additionally, the 3D representation can be dynamically updated as new transmission data is received, allowing for real-time visualization. The invention is particularly useful in scenarios where high-fidelity 3D reconstructions are required from multi-source or multi-dimensional transmission data.
20. The computing system of claim 17, wherein the reconstruction system applies shading and/or color data onto the 3D representation, using the calibration data to map portions of the color information to the 3D representation.
This invention relates to computing systems for generating and processing three-dimensional (3D) representations of objects, particularly focusing on enhancing the visual accuracy of these representations. The system addresses the challenge of accurately mapping color and shading data onto a 3D model to improve realism and fidelity. The reconstruction system processes input data, such as images or sensor measurements, to generate a 3D representation of an object. To enhance this representation, the system applies shading and color data, which may be derived from the same input data or additional sources. Calibration data is used to ensure precise alignment between the color information and the 3D model, correcting distortions or misalignments that could arise during data acquisition or processing. The calibration data may include geometric corrections, lighting conditions, or sensor-specific adjustments. By applying this calibrated data, the system accurately maps color and shading to the correct portions of the 3D representation, resulting in a more visually accurate and realistic output. This approach is particularly useful in applications like 3D scanning, medical imaging, or virtual reality, where visual fidelity is critical. The system may also include preprocessing steps to refine the input data before reconstruction and post-processing steps to further enhance the final 3D representation.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
May 3, 2023
April 23, 2024
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.