Described herein are platforms, systems, media, and methods for measuring a space by launching an active augmented reality (AR) session on a device comprising a camera and at least one processor; calibrating the AR session by establishing a fixed coordinate system, receiving a position and orientation of one or more horizontal or vertical planes in the space in reference to the fixed coordinate system, and receiving a position and orientation of the camera in reference to the fixed coordinate system; constructing a backing model; providing an interface allowing a user to capture at least one photo of the space during the active AR session; extracting camera data from the AR session for the at least one photo; extracting the backing model from the AR session; and storing the camera data and the backing model in association with the at least one photo.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
3. The system of claim 2, wherein the user identifies screen coordinates by tapping on a touchscreen, tapping and dragging on a touch screen, clicking with a pointing device, or clicking and dragging with a pointing device.
This invention relates to a system for user interaction with a graphical user interface (GUI) to select or manipulate elements on a display. The system enables users to identify screen coordinates through various input methods, including tapping on a touchscreen, tapping and dragging on a touchscreen, clicking with a pointing device, or clicking and dragging with a pointing device. The system processes these inputs to determine the corresponding screen coordinates, allowing for precise interaction with displayed elements. The input methods accommodate different user preferences and device capabilities, ensuring flexibility in how users engage with the interface. The system may be used in applications requiring precise coordinate selection, such as graphic design, mapping, or touch-based interfaces, where accurate input is essential for effective interaction. By supporting multiple input modalities, the system enhances usability across different devices and user scenarios.
4. The system of claim 2, wherein the measurements and annotations are stored in association with the at least one photo as metadata associated with the at least one photo.
This invention relates to a system for capturing, storing, and managing measurements and annotations associated with photographs, particularly in applications such as construction, real estate, or inspection. The system addresses the challenge of efficiently documenting and retrieving precise measurements and notes linked to specific images, ensuring accuracy and accessibility for later review or analysis. The system includes a device capable of capturing photographs and recording measurements and annotations related to the photographed objects or scenes. These measurements and annotations are stored as metadata directly associated with the photo files, allowing them to be embedded within the image data or linked externally in a structured format. This integration ensures that the contextual information remains intact and easily retrievable alongside the visual data, eliminating the need for separate documentation. The system may also include features for organizing and retrieving the annotated photos based on the stored metadata, such as filtering or searching by measurement values, annotation content, or other criteria. This functionality enhances workflow efficiency by enabling quick access to relevant data without manual cross-referencing. The system may further support collaboration by allowing multiple users to contribute or access the annotated photos and their associated metadata, facilitating team-based projects or inspections. By storing measurements and annotations as metadata, the system ensures that critical information is preserved in a standardized, searchable format, reducing errors and improving data management in fields where precise documentation is essential.
5. The system of claim 2, wherein the measurements and annotations are stored in association with the at least one photo by a key, token, or link.
This invention relates to a system for capturing, storing, and managing measurements and annotations associated with photographs, particularly in fields such as construction, real estate, or inspection where precise documentation is critical. The system addresses the challenge of maintaining accurate and organized records by linking measurements and annotations directly to the corresponding photographs, ensuring data integrity and easy retrieval. The system includes a device, such as a smartphone or tablet, equipped with a camera and sensors to capture images and measurements. Users can take photos of a site or object and simultaneously record measurements (e.g., dimensions, distances) and annotations (e.g., notes, labels) using the device. These measurements and annotations are then stored in a database or cloud storage system, where they are linked to the corresponding photo via a unique identifier, such as a key, token, or hyperlink. This linkage ensures that the data remains associated with the correct image, even if files are moved or shared. The system may also include features for organizing and retrieving stored data, such as search functionality or filtering options based on measurement values or annotation content. Additionally, the system may support collaboration by allowing multiple users to access and contribute to the same dataset, with permissions and version control to manage edits. The invention improves efficiency by reducing manual record-keeping errors and streamlining the process of documenting and reviewing site conditions.
6. The system of claim 2, wherein the first processing device or the second processing device is further configured to provide an interface allowing a user to edit the screen coordinates identified on the at least one photo.
A system for processing images to identify and edit screen coordinates on captured photos. The system addresses the challenge of accurately detecting and modifying screen content within images, which is useful for applications like augmented reality, computer vision, and digital content analysis. The system includes at least one camera for capturing photos and at least two processing devices. The first processing device analyzes the captured photos to identify screen coordinates, such as the boundaries or regions of a displayed screen within the image. The second processing device then processes these identified coordinates to extract or analyze the screen content. The system further includes an interface that allows a user to manually edit the identified screen coordinates, enabling corrections or refinements to the automated detection process. This editing capability ensures accuracy, particularly in cases where the automated detection may produce errors due to lighting conditions, reflections, or other visual distortions. The system may also include a display for presenting the edited coordinates and a storage device for retaining the processed data. The ability to edit screen coordinates enhances the system's flexibility and usability in various applications requiring precise screen content extraction or analysis.
9. The system of claim 1, wherein the first processing device or the second processing device is further configured to allow the user to make corrections to the backing model based on measurements taken in the at least one photo.
This invention relates to a system for creating and refining 3D models using photographic images. The system addresses the challenge of accurately generating 3D models from 2D photographs, particularly when the initial model (backing model) may contain errors or inaccuracies. The system includes at least two processing devices that work together to process photographic images and generate a 3D model. The first processing device captures or receives at least one photo of an object or scene, while the second processing device processes the photo to create an initial 3D model. The system then allows a user to compare the 3D model with the original photo and make corrections to the model based on measurements taken from the photo. This ensures the final 3D model accurately reflects the real-world dimensions and features of the photographed object or scene. The system may also include a display device to show the 3D model and the photo side by side, facilitating user corrections. The ability to refine the model based on direct measurements from the photo improves accuracy and reduces errors in the final 3D representation.
10. The system of claim 1, wherein the first processing device or the second processing device is further configured to transmit the stored camera data, the stored backing model, and the at least one photo.
A system for processing and analyzing visual data includes a first processing device and a second processing device. The first processing device captures and stores camera data, such as images or video, and generates a backing model representing the spatial or structural characteristics of the captured environment. The second processing device processes the stored camera data and the backing model to generate at least one photo, which may be a synthesized image or a processed version of the captured data. The system is designed to enhance visual data analysis, reconstruction, or rendering by leveraging the backing model to improve accuracy or efficiency in generating the final output. The stored camera data, backing model, and generated photo can be transmitted for further processing, storage, or display. This system is useful in applications requiring detailed environmental reconstruction, such as augmented reality, 3D modeling, or autonomous navigation, where accurate spatial representation and efficient data handling are critical. The transmission of these components allows for distributed processing or collaboration across multiple devices.
11. The system of claim 1, wherein the camera data comprises one or more of: projection matrix, view matrix, view port, camera position, view angle, scale factor.
A system for capturing and processing camera data in a three-dimensional environment includes components for generating and managing visual information. The system captures camera data, which may include a projection matrix, view matrix, viewport, camera position, view angle, and scale factor. The projection matrix defines how three-dimensional points are projected onto a two-dimensional plane, while the view matrix determines the camera's orientation and position in the scene. The viewport specifies the visible area of the rendered image, and the camera position indicates the location of the camera in the three-dimensional space. The view angle defines the field of view, and the scale factor adjusts the size of objects relative to the camera. This data is used to render and manipulate visual content accurately within the system, ensuring proper perspective and alignment in the displayed output. The system may also include additional components for processing and displaying the rendered content, such as graphics processing units or display interfaces. The captured camera data allows for dynamic adjustments to the viewing parameters, enabling real-time modifications to the visual representation of the environment. This approach enhances the accuracy and flexibility of three-dimensional rendering, particularly in applications requiring precise spatial relationships, such as virtual reality, augmented reality, or computer-aided design.
13. The system of claim 12, wherein the first processing device is further configured to apply one or more deep learning models to identify one or more seams between the floor and virtual walls to refine the positions of the corners and the floorplan.
This invention relates to a system for refining floorplan representations in virtual environments, particularly for applications in augmented reality, robotics, or digital mapping. The system addresses the challenge of accurately detecting and refining the boundaries between floors and virtual walls to improve the precision of floorplan layouts. The system includes a first processing device that processes sensor data, such as images or depth scans, to generate an initial floorplan representation. This initial representation includes detected corners and edges that define the layout of walls and floors. To enhance accuracy, the system applies one or more deep learning models to analyze the sensor data and identify seams—transition points or boundaries—between the floor and virtual walls. These seams are used to refine the positions of the corners and the overall floorplan, ensuring that the virtual representation aligns more closely with the physical environment. The deep learning models may be trained to recognize patterns or features in the sensor data that indicate the presence of seams, such as texture changes, depth discontinuities, or geometric inconsistencies. By refining the floorplan based on these seams, the system improves the accuracy of virtual environment mapping, which is critical for applications like robot navigation, augmented reality overlays, or digital twin simulations. The refined floorplan can then be used for further processing, such as object placement, path planning, or environmental interaction.
14. The system of claim 12, wherein the first processing device is further configured to provide an interface allowing a user to rectify the floorplan by enforcing angles of all segments of the floorplan to fall into a predetermined set of angles.
This invention relates to a system for processing and rectifying floorplans, particularly in architectural or construction applications. The system addresses the challenge of ensuring geometric accuracy in floorplan representations, where irregular or non-standard angles can lead to errors in design, construction, or analysis. The system includes a first processing device configured to analyze a floorplan and identify segments within the floorplan. These segments represent walls, boundaries, or other structural elements. The system further includes a second processing device that determines the angles between these segments. The first processing device then compares these angles to a predefined set of acceptable angles, such as 90 degrees, 45 degrees, or other standard construction angles. If any segment angle deviates from the predefined set, the system provides an interface that allows a user to rectify the floorplan by adjusting the angles to conform to the predetermined set. This ensures that the floorplan adheres to standard geometric constraints, improving accuracy and compatibility with construction tools or software. The system may also include a display device to visualize the floorplan and the rectification process, as well as a storage device to retain the original and adjusted floorplan data. The rectification process may involve automatic adjustments or user-guided corrections to ensure the floorplan meets design specifications.
15. The system of claim 12, wherein the first processing device is further configured to provide an interface allowing a user to re-order the positions of corners of the floor of the space to create the desired floorplan geometry.
This invention relates to a system for designing and visualizing floorplans of a space. The system addresses the challenge of creating accurate and customizable floorplan geometries, particularly for spaces with irregular or complex layouts. The system includes a first processing device that generates a floorplan representation of the space, where the floorplan is defined by a set of corners representing the boundaries of the floor. The system allows a user to interactively adjust the positions of these corners to modify the floorplan geometry. The first processing device provides an interface that enables the user to re-order the positions of the corners, allowing for precise control over the shape and dimensions of the floorplan. This functionality ensures that the floorplan accurately reflects the desired layout of the space, whether it is a simple rectangular room or a more complex, multi-sided configuration. The system may also include additional features, such as generating a three-dimensional model of the space based on the floorplan, or integrating the floorplan with other design tools for further customization. The invention improves the efficiency and flexibility of floorplan design by providing an intuitive and user-friendly method for adjusting the geometry of the floorplan.
16. The system of claim 1, wherein the first processing device or the second processing device is further configured to convert the at least one photo to a transmittable format.
The invention relates to a system for processing and transmitting digital images, particularly in environments where image data must be converted into a transmittable format. The system includes at least two processing devices that handle image data, such as capturing, storing, or transmitting photos. A key challenge addressed by this invention is ensuring that images can be efficiently converted into a format suitable for transmission, especially when dealing with large or complex image files that may not be natively compatible with transmission protocols or storage constraints. The system includes a first processing device and a second processing device, each capable of performing image processing tasks. One of these devices is further configured to convert at least one photo into a transmittable format. This conversion process may involve compressing the image, adjusting resolution, or applying other transformations to ensure compatibility with transmission standards or network limitations. The system may also include additional components, such as sensors or storage units, that interact with the processing devices to facilitate image capture, storage, or transmission. The invention is particularly useful in applications where images must be transmitted over networks with limited bandwidth or where storage space is constrained. By converting images into a transmittable format, the system ensures that data can be efficiently shared or stored without losing critical information. The conversion process may be automated or triggered based on specific conditions, such as file size or transmission requirements.
17. The system of claim 1, wherein the camera data and the backing model are stored in a structured or semi-structured data format.
The invention relates to a system for processing and storing camera data and a backing model in a structured or semi-structured data format. The system addresses the challenge of efficiently managing and analyzing large volumes of camera data, particularly in applications such as surveillance, autonomous vehicles, or industrial automation, where real-time processing and structured data storage are critical. The backing model, which may include machine learning models, algorithms, or reference data, is used to enhance the analysis of camera data, such as object detection, tracking, or scene understanding. By storing both the camera data and the backing model in a structured or semi-structured format, the system enables easier integration with databases, analytics tools, and other software systems. This structured approach facilitates faster querying, indexing, and retrieval of data, improving overall system performance and scalability. The system may also include components for preprocessing camera data, such as noise reduction, frame alignment, or feature extraction, to optimize the data before storage. Additionally, the system may support real-time or batch processing of camera data, depending on the application requirements. The structured or semi-structured data format ensures compatibility with various data processing frameworks and allows for seamless interoperability with other systems. This invention enhances the efficiency and reliability of camera data analysis in diverse technical domains.
18. The system of claim 1, wherein the camera data and the backing model are stored in an encrypted format.
A system for processing and storing camera data and a backing model in an encrypted format is disclosed. The system addresses security concerns in data handling, particularly in applications where sensitive visual information or proprietary models are involved. The camera data, which may include images or video streams captured by one or more cameras, is encrypted to prevent unauthorized access during storage or transmission. Similarly, the backing model, which could be a machine learning model, algorithm, or reference dataset used to process or analyze the camera data, is also encrypted. This dual encryption ensures that both the raw input data and the analytical tools remain secure throughout their lifecycle. The system may include components for encryption, decryption, and secure storage, as well as interfaces for authorized users or systems to access the encrypted data and models when needed. By encrypting both the camera data and the backing model, the system mitigates risks of data breaches, intellectual property theft, and unauthorized use of sensitive information. This approach is particularly useful in fields such as surveillance, healthcare, or financial services, where data privacy and security are critical.
19. The system of claim 1, wherein the capture of the at least one photo of the space during the active AR session is triggered by a local user present in the space and with the first processing device.
This invention relates to augmented reality (AR) systems designed to capture and process images of a physical space during an active AR session. The system addresses the challenge of dynamically documenting real-world environments in AR applications, where users may need to record visual data for later analysis, sharing, or integration with digital content. The system includes a first processing device, such as a smartphone or AR headset, equipped with a camera and capable of running an AR application. During an active AR session, the device captures at least one photo of the surrounding space. The capture is triggered by a local user physically present in the space, ensuring that the image is taken at a relevant moment, such as when the user encounters an object of interest or needs to document a specific view. The system may also include additional processing devices or sensors to enhance the AR experience, such as depth sensors or motion trackers, which help contextualize the captured images within the AR environment. The captured photos may be used for various purposes, including mapping the space, generating 3D models, or overlaying digital content onto the real-world scene. The system ensures that the images are taken in real-time, providing an accurate representation of the space as it appears during the AR session. This functionality is particularly useful in applications like interior design, remote assistance, or gaming, where real-world context is critical. The invention improves the usability of AR systems by enabling users to document their environment seamlessly while interacting with digital overlays.
20. The system of claim 1, wherein the capture of the at least one photo of the space during the active AR session is triggered by a remote user not present in the space.
A system for augmented reality (AR) captures photos of a physical space during an active AR session, where the capture is triggered by a remote user not physically present in the space. The system includes an AR device worn by a local user in the space, which displays AR content overlaid on the real-world environment. The AR device communicates with a remote computing device operated by the remote user, allowing the remote user to initiate photo capture of the space as seen through the AR device. The captured photos are transmitted to the remote computing device, enabling the remote user to view the space and the AR content from the perspective of the local user. This system facilitates collaboration between remote and local users, where the remote user can provide guidance or feedback based on the captured images. The AR device may also include sensors to track the local user's movements and adjust the AR content accordingly. The system ensures real-time interaction, allowing the remote user to trigger photo capture at specific moments during the AR session. This technology addresses the challenge of remote collaboration in AR environments, where physical presence is not possible, by enabling remote users to visually assess the local environment and AR content in real time.
21. The system of claim 1, wherein the first processing device or the second processing device is further configured to provide an interface allowing a user to edit the position or orientation of the one or more horizontal or vertical planes in the space in reference to the fixed coordinate system.
This invention relates to a system for visualizing and manipulating spatial data, particularly in a three-dimensional environment. The system addresses the challenge of accurately representing and adjusting spatial relationships in a fixed coordinate system, which is critical for applications such as augmented reality, robotics, and industrial design. The system includes at least two processing devices that work together to generate and display one or more horizontal or vertical planes within a defined space. These planes are aligned with a fixed coordinate system, ensuring precise spatial referencing. The system allows users to interact with these planes, enabling adjustments to their position or orientation relative to the fixed coordinate system. This functionality is provided through an interface that facilitates intuitive manipulation of the planes, enhancing usability and accuracy in spatial data representation. The system may also include additional features, such as the ability to generate and display multiple planes simultaneously, ensuring comprehensive spatial coverage. The processing devices may further process sensor data to refine plane positioning, improving accuracy in dynamic environments. The interface may support various input methods, including touch, gesture, or voice commands, to accommodate different user preferences and workflows. By enabling precise and user-friendly manipulation of spatial planes, this system enhances the accuracy and efficiency of tasks requiring spatial data visualization and interaction.
22. The system of claim 1, wherein the first processing device or the second processing device is further configured to provide an interface allowing a user to adjust a scale of a floorplan and 3D model by adjusting a virtual floor-plane height incrementally such that modeled object dimensions and aspect ratios match those of a known physical size of the space.
This invention relates to a system for visualizing and adjusting floorplans and 3D models of physical spaces. The system addresses the challenge of accurately scaling digital representations to match real-world dimensions, ensuring that modeled objects maintain correct proportions relative to known physical measurements of the space. The system includes at least two processing devices that generate and display a floorplan and a corresponding 3D model of a space. A user interface allows adjustments to the scale of both representations by modifying a virtual floor-plane height incrementally. This adjustment ensures that the dimensions and aspect ratios of objects in the model align with the known physical size of the space. The system may also include a display device for presenting the floorplan and 3D model, as well as a memory for storing data related to the space and the models. The incremental adjustment of the virtual floor-plane height enables precise scaling, preventing distortion of object proportions while maintaining accuracy relative to real-world measurements. This feature is particularly useful in architectural, interior design, or construction applications where precise scaling is critical for planning and visualization. The system ensures that digital models remain true to physical dimensions, improving decision-making and reducing errors in spatial planning.
23. The system of claim 1, wherein the first processing device or the second processing device is further configured to utilize data collected from one or more deep learning models to correct scale or drift in the backing model.
The invention relates to a system for improving the accuracy of a backing model, which is a predictive or analytical model used in various applications such as robotics, autonomous systems, or data analysis. The backing model may suffer from scale or drift issues, where its predictions become less accurate over time due to changes in input data distributions or environmental conditions. To address this, the system includes at least two processing devices that collect data from one or more deep learning models. These deep learning models are trained to detect and correct errors in the backing model's outputs, such as discrepancies in scale (e.g., magnitude of predictions) or drift (e.g., systematic biases). The processing devices use the corrected data to adjust the backing model, ensuring its predictions remain reliable. The deep learning models may be trained on historical data, real-time sensor inputs, or other relevant datasets to provide continuous or periodic corrections. This approach enhances the robustness and longevity of the backing model in dynamic environments.
24. The system of claim 1, wherein the first processing device, the second processing device, or both are further configured to provide an interface allowing a user to model ceiling geometries from the at least one photo of the space by hit-testing and identification of ceiling planes, facets, and boundaries.
This invention relates to a system for modeling ceiling geometries from photographic images of interior spaces. The system addresses the challenge of accurately capturing and digitizing ceiling structures, which is often complex due to varying shapes, angles, and lighting conditions in real-world environments. The system includes at least one processing device that processes images of a space to extract ceiling geometry data. The processing device identifies ceiling planes, facets, and boundaries within the images using hit-testing techniques, which involve detecting intersections between user inputs and the image data to refine the model. The system allows users to interactively adjust and refine the detected ceiling structures through an interface, ensuring accurate representation of the space. The system may also include a second processing device that collaborates with the first to enhance processing efficiency or distribute computational tasks. Both devices can be configured to support the modeling interface, enabling users to manually verify and correct automated detections. The interface facilitates precise modeling by allowing users to define and adjust ceiling planes, facets, and boundaries based on visual feedback from the images. This approach improves the accuracy and usability of ceiling modeling in applications such as architectural design, virtual reality, and interior visualization, where precise spatial data is critical.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
July 29, 2021
December 13, 2022
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.