Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A non-transitory computer-readable storage medium storing a program that causes a computer to execute a process, the process comprising: estimating, by a first detection device disposed in a real space, a first three-dimensional position of a wearable display device and a first arrangement of the wearable display device including a screen configured to display an image indicating a virtual space including a virtual target object corresponding to a physical object located in the real space, the first detection device tracking the first three-dimensional position and the first arrangement; estimating, by a second detection device disposed on the wearable display device, a second three-dimensional position of the physical object and a second arrangement of the physical object, the second detection device tracking an area having a specified feature corresponding to the physical object in the real space; estimating a third three-dimensional position of a specified body part, in the real space, of a person wearing the wearable display device; determining whether a positional relationship in the real space between the physical object and the specified body part satisfies a predetermined criteria based on the second three-dimensional position and the third three-dimensional position; displaying, on the screen, the virtual target object with a display position and a display arrangement determined based on the second three-dimensional position and the second arrangement when the positional relationship satisfies the predetermined criteria; and displaying, on the screen, the virtual target object with a display position and a display arrangement determined based on the first three-dimensional position and the first arrangement when the positional relationship does not satisfy the predetermined criteria.
This invention relates to augmented reality (AR) systems that integrate virtual objects with real-world environments. The system addresses the challenge of accurately aligning virtual objects with physical objects in real space, particularly when tracking conditions vary. The invention uses multiple detection devices to improve positional accuracy and user interaction. A wearable display device presents a virtual space containing a virtual target object corresponding to a physical object in the real world. A first detection device, placed in the real space, tracks the wearable display's three-dimensional position and orientation, while a second detection device on the wearable display tracks the physical object's position and orientation by identifying a feature associated with the object. Additionally, the system estimates the position of a specific body part of the user wearing the display. The system determines whether the physical object and the user's body part meet a predefined positional relationship based on their tracked positions. If the criteria are satisfied, the virtual target object is displayed in a position and orientation aligned with the physical object's tracked data. If not, the virtual object is displayed based on the wearable display's tracked position and orientation. This approach ensures accurate virtual-to-real alignment while adapting to tracking conditions. The invention enhances AR experiences by dynamically adjusting virtual object placement based on real-world interactions.
2. The non-transitory computer-readable storage medium according to claim 1 , wherein the wearable display device is a head mount display device.
A wearable display device, specifically a head-mounted display (HMD), is used to present visual information to a user. The device includes a display module that generates and projects images or data directly into the user's field of view. The system also incorporates a sensor module that detects environmental or user-specific data, such as motion, orientation, or biometric information. A processing module analyzes this data to determine the user's context, such as their physical activity, location, or physiological state. Based on this analysis, the processing module dynamically adjusts the content displayed on the HMD, ensuring the information remains relevant and useful. For example, if the user is exercising, the display may show real-time fitness metrics, while in a work environment, it may prioritize task-related data. The system may also include a communication module to transmit or receive data from external sources, enabling real-time updates or remote monitoring. The HMD is designed to be lightweight, ergonomic, and capable of long-term wear, ensuring user comfort and practicality. This technology addresses the need for adaptive, context-aware displays that enhance user experience by providing timely and personalized information.
3. The non-transitory computer-readable storage medium according to claim 1 , wherein the estimating the second three-dimensional position and the second arrangement includes: detecting a three-dimensional position of a marker and an arrangement of the marker placed within the real space in association with the physical object; and estimating the second three-dimensional position and the second arrangement based on the three-dimensional position of the marker and the arrangement of the marker.
This invention relates to augmented reality (AR) systems that overlay virtual objects onto real-world environments. A key challenge in AR is accurately tracking and aligning virtual content with physical objects in real space to ensure proper positioning and orientation. The invention addresses this by using markers placed in association with physical objects to improve the accuracy of virtual object placement. The system involves a non-transitory computer-readable storage medium storing instructions for estimating the position and arrangement of a virtual object relative to a physical object in real space. The process includes detecting a three-dimensional position and arrangement of a marker placed near or on the physical object. The marker's detected position and arrangement are then used to estimate the three-dimensional position and orientation of the virtual object relative to the physical object. This ensures that the virtual object is accurately aligned with the physical object in the AR environment. The marker-based approach enhances tracking precision, reducing misalignment issues common in AR applications. By leveraging the marker's known position and arrangement, the system can dynamically adjust the virtual object's placement in real time, improving user experience in applications such as gaming, industrial training, or virtual prototyping. The method is particularly useful in scenarios where traditional tracking methods, like camera-based or sensor-based tracking, may be less reliable.
4. The non-transitory computer-readable storage medium according to claim 1 , wherein the predetermined criteria indicate that a distance between the physical object and the specified body part is equal to or less than a predetermined distance.
This invention relates to a system for detecting interactions between a physical object and a specified body part of a user, such as a hand or finger, using computer vision techniques. The system addresses the challenge of accurately determining when a user's body part comes into close proximity with or contacts a physical object in real-time, which is useful for applications like virtual reality, augmented reality, or interactive displays. The system processes image data captured by one or more cameras to identify the physical object and the specified body part. It then calculates the distance between the two and compares this distance to a predetermined threshold. If the distance is equal to or less than the threshold, the system determines that an interaction has occurred. This allows for precise detection of interactions without requiring physical contact, enabling more natural and intuitive user interfaces. The predetermined distance threshold can be adjusted based on the specific application, ensuring flexibility in defining what constitutes an interaction. The system may also incorporate additional criteria, such as the duration of proximity or the relative motion between the body part and the object, to refine interaction detection. This approach improves accuracy and reduces false positives in dynamic environments.
5. The non-transitory computer-readable storage medium according to claim 4 , wherein the determining determines whether the positional relationship satisfies the predetermined criteria by performing image processing of an image that includes the physical object and the specified body part.
This invention relates to a computer-implemented system for analyzing the positional relationship between a physical object and a specified body part of a user. The system addresses the challenge of accurately assessing whether the spatial arrangement between an object and a body part meets predefined criteria, which is critical in applications such as ergonomics, medical monitoring, or human-computer interaction. The system processes an image containing both the physical object and the specified body part to evaluate their positional relationship. Image processing techniques are used to extract relevant features, such as the relative positions, orientations, or distances between the object and the body part. These features are then compared against predetermined criteria to determine if the positional relationship is acceptable or requires adjustment. The system may also include a display device to provide feedback to the user, such as visual or auditory signals indicating whether the criteria are satisfied. The invention may further include a sensor, such as a camera or depth sensor, to capture the image of the physical object and the body part. The system may also incorporate a processor to execute the image processing algorithms and a memory to store the predetermined criteria and processed data. The overall goal is to provide real-time or near-real-time analysis of the positional relationship, enabling dynamic feedback and adjustments to improve user experience or safety.
6. The non-transitory computer-readable storage medium according to claim 5 , wherein the wearable display device includes a camera device that captures the image; and wherein the displaying displays the virtual target object and a virtual object other than the virtual target object superimposing with the image obtained by the camera device.
This invention relates to augmented reality (AR) systems, specifically wearable display devices that overlay virtual objects onto real-world images captured by a camera. The technology addresses the challenge of enhancing user interaction with virtual objects in AR environments by improving the display of multiple virtual objects in relation to real-world imagery. The wearable display device includes a camera that captures real-world images. The system then displays virtual objects, including a primary virtual target object and at least one additional virtual object, superimposed onto the captured real-world image. This allows users to see both virtual and real-world elements in a single, cohesive view, enhancing spatial awareness and interaction. The invention ensures that the virtual objects are accurately aligned with the real-world environment, providing a seamless AR experience. The system may also include additional features, such as tracking the user's position or orientation to maintain proper alignment of the virtual objects with the real-world scene. This technology is particularly useful in applications like gaming, navigation, or industrial training, where users need to interact with virtual elements in a real-world context.
7. The non-transitory computer-readable storage medium according to claim 6 , wherein the image obtained by the camera device includes a part of the real space corresponding to a field of view of the wearable display device.
This invention relates to augmented reality (AR) systems, specifically improving the alignment of virtual content with real-world environments. The problem addressed is the misalignment of virtual objects in AR displays due to inaccuracies in tracking the user's position and orientation relative to the real world. The solution involves a non-transitory computer-readable storage medium storing instructions that, when executed, perform a method for enhancing AR display accuracy. The method includes obtaining an image of the real space from a camera device, where the image captures a portion of the environment visible to the user through a wearable display device. The system then processes this image to determine the user's position and orientation in real space, ensuring that virtual content is accurately overlaid on the correct real-world locations. The method also involves adjusting the display of virtual content based on the processed image data to correct any misalignment. This approach improves the realism and usability of AR applications by dynamically compensating for tracking errors, ensuring that virtual objects appear correctly positioned relative to the physical environment. The invention is particularly useful in AR headsets and other wearable displays where precise spatial alignment is critical for user experience.
8. The non-transitory computer-readable storage medium according to claim 4 , wherein the determining determines whether the positional relationship satisfies the predetermined criteria by using a sensor that is attached to at least one of the physical object and the specified body part and that detects proximity of the specified body part to the physical object.
This invention relates to a system for detecting and evaluating the positional relationship between a physical object and a specified body part of a user. The system addresses the challenge of accurately determining whether the body part is in a desired proximity to the object, which is critical for applications such as ergonomic monitoring, safety compliance, or interactive user interfaces. The system includes a sensor attached to either the physical object, the specified body part, or both. The sensor detects proximity between the body part and the object, generating data that is processed to determine whether the positional relationship meets predetermined criteria. The criteria may include distance thresholds, relative orientation, or other spatial conditions. The sensor may use technologies such as infrared, ultrasonic, or capacitive sensing to measure proximity. The system may also include a processor that analyzes the sensor data to assess compliance with the criteria. If the criteria are met, the system may trigger an action, such as providing feedback to the user, adjusting the object's position, or logging the event. The invention ensures precise and reliable detection, improving accuracy in applications where spatial relationships are critical.
9. The non-transitory computer-readable storage medium according to claim 8 , wherein the displaying displays the specified body part of the real space without concealing the virtual target object when the image obtained by the camera device includes the specified body part.
This invention relates to augmented reality (AR) systems that overlay virtual objects onto real-world images captured by a camera. A common problem in AR is ensuring that virtual objects do not obscure important real-world elements, such as a user's body parts, which can disrupt the user experience. The invention addresses this by dynamically adjusting the display of virtual objects to avoid concealing specified body parts in the captured image. The system includes a camera device that captures images of a real space, a display device that presents augmented reality content, and a processor that processes the captured images. The processor identifies specified body parts, such as hands or limbs, within the captured images. When the image includes a specified body part, the system displays the virtual target object in a way that does not conceal that body part. This ensures that the user can see both the virtual object and the body part simultaneously, improving interaction and usability in AR applications. The invention may also involve tracking the position and orientation of the body part in real time to dynamically adjust the virtual object's display. This prevents the virtual object from overlapping the body part as the user moves, maintaining a clear and intuitive AR experience. The system may further include user input mechanisms to define or modify the specified body parts, allowing customization based on user preferences or application requirements. This approach enhances AR applications in gaming, training, or productivity tools where body part visibility is critical.
10. The non-transitory computer-readable storage medium according to claim 4 , wherein the second detection device is a stereo camera device.
A system for object detection in a vehicle environment uses a stereo camera device to enhance detection accuracy. The stereo camera device captures depth information by analyzing disparities between images from two or more cameras, allowing precise three-dimensional positioning of objects. This depth data is combined with other sensor inputs, such as radar or lidar, to improve object detection and tracking. The system processes the stereo camera data to identify objects, estimate their distance, and determine their movement patterns. By integrating multiple sensor modalities, the system achieves robust detection even in challenging conditions like low visibility or dynamic environments. The stereo camera's depth perception helps distinguish between overlapping objects and reduces false positives, ensuring reliable performance for applications like autonomous driving or advanced driver-assistance systems. The system may also use the stereo camera to verify or refine detections from other sensors, improving overall accuracy and reliability. This approach leverages the strengths of different sensor types to create a comprehensive perception system for vehicle safety and navigation.
11. The non-transitory computer-readable storage medium according to claim 1 , wherein displaying, on the screen, the virtual target object with the display position and display arrangement determined based on the second three-dimensional position and the second arrangement reduces a difference between visual perception and tactile perception of the person viewing the screen while the physical object is being manipulated by the specified body part of the person.
This invention relates to virtual reality (VR) or augmented reality (AR) systems that enhance the alignment between visual and tactile feedback when a user interacts with physical objects. The problem addressed is the mismatch between what a user sees on a screen and the physical sensations they experience when manipulating real-world objects, which can cause disorientation or reduced immersion. The system involves a computer-readable storage medium containing instructions for a VR/AR device. The device tracks the three-dimensional position and orientation of a physical object being manipulated by a user's body part (e.g., hand or finger). Based on this tracking data, the system determines the optimal display position and arrangement of a corresponding virtual target object on the screen. The virtual object is then rendered in a way that minimizes the discrepancy between the visual representation and the tactile feedback from the physical object. This synchronization improves user experience by making interactions feel more natural and intuitive. The system may also adjust the virtual object's appearance or behavior dynamically to further reduce perceptual mismatches, such as compensating for delays in tracking or rendering. The goal is to create a seamless integration between the virtual and physical worlds, enhancing immersion and usability in applications like gaming, training simulations, or industrial design.
12. A computer-implemented method comprising: estimating, by a first detection device disposed in a real space, a first three-dimensional position of a wearable display device and a first arrangement of the wearable display device including a screen configured to display an image indicating a virtual space including a virtual target object corresponding to a physical object located in the real space, the first detection device tracking the first three-dimensional position and the first arrangement; estimating, by a second detection device disposed on the wearable display device, a second three-dimensional position of the physical object and a second arrangement of the physical object, the second detection device tracking an area having a specified feature corresponding to the physical object in the real space; estimating a third three-dimensional position of a specified body part, in the real space, of a person wearing the wearable display device; determining whether a positional relationship in the real space between the physical object and the specified body part satisfies a predetermined criteria based on the second three-dimensional position and the third three-dimensional position; displaying, on the screen, the virtual target object with a display position and a display arrangement determined based on the second three-dimensional position and the second arrangement when the positional relationship satisfies the predetermined criteria; and displaying, on the screen, the virtual target object with a display position and a display arrangement determined based on the first three-dimensional position and the first arrangement when the positional relationship does not satisfy the predetermined criteria.
This invention relates to augmented reality (AR) systems that integrate virtual objects with real-world environments. The system addresses the challenge of accurately aligning virtual objects with physical objects in real space, particularly when the user's movement or the physical object's position changes. The method involves multiple detection devices to track both the wearable display device and physical objects in the real space. A first detection device, placed in the real space, estimates the three-dimensional position and orientation of the wearable display device, which includes a screen displaying a virtual space with a virtual target object corresponding to a physical object. A second detection device, mounted on the wearable display, tracks the physical object's position and orientation by identifying a specified feature in the real space. Additionally, the system estimates the position of a specified body part of the user wearing the display. The system determines whether the positional relationship between the physical object and the user's body part meets predetermined criteria. If the criteria are satisfied, the virtual target object is displayed based on the physical object's tracked position and orientation. If not, the virtual object is displayed based on the wearable device's tracked position and orientation. This ensures accurate alignment of virtual and physical objects while adapting to user movement and environmental changes.
13. The computer-implemented method according to claim 12 , wherein the estimating the second three-dimensional position and the second arrangement includes: detecting a three-dimensional position of a marker and an arrangement of the marker placed within the real space in association with the physical object; and estimating the second three-dimensional position and the second arrangement based on the three-dimensional position of the marker and the arrangement of the marker.
This invention relates to computer-implemented methods for estimating the position and arrangement of physical objects in a real space using markers. The problem addressed is accurately determining the spatial relationship of objects in three-dimensional space, which is challenging due to factors like occlusion, lighting variations, and sensor limitations. The method involves detecting a marker placed in association with a physical object within the real space. The marker's three-dimensional position and its arrangement (e.g., orientation or alignment) are identified. Based on this detected information, the method estimates the second three-dimensional position and arrangement of the object. This approach leverages the marker as a reference point to improve positional accuracy and reduce errors in object tracking or localization. The method may also include additional steps such as capturing images or sensor data of the real space, processing the data to identify the marker, and using the marker's known properties (e.g., size, shape, or pattern) to refine the estimation. The marker can be a physical tag, a visual pattern, or a sensor-based beacon, depending on the application. This technique is useful in augmented reality, robotics, industrial automation, and other fields requiring precise object positioning.
14. The computer-implemented method according to claim 12 , wherein the predetermined criteria indicates that a distance between the physical object and the specified body part is equal to or less than a predetermined distance.
This invention relates to computer-implemented methods for detecting interactions between physical objects and specified body parts, such as hands or fingers, in a virtual or augmented reality environment. The problem addressed is accurately determining when a physical object is in close proximity to a user's body part, which is crucial for applications like virtual reality training, gaming, or assistive technologies. The method involves tracking the position of a physical object and a specified body part in a three-dimensional space. The system calculates the distance between the object and the body part and compares it to a predetermined threshold distance. If the distance is equal to or less than this threshold, the system identifies this as an interaction event. This allows for precise detection of when a user's body part is near or touching the object, enabling real-time feedback or triggering specific actions in the virtual environment. The predetermined distance threshold can be adjusted based on the application, ensuring flexibility for different use cases. For example, in a surgical training simulation, the threshold might be set to detect when a surgeon's hand is within a few millimeters of a virtual scalpel. The method improves accuracy in interaction detection, reducing false positives and negatives, and enhances user experience in immersive environments.
15. The computer-implemented method according to claim 14 , wherein the determining determines whether the positional relationship satisfies the predetermined criteria by performing image processing of a captured image that includes the physical object and the specified body part.
This invention relates to computer-implemented methods for analyzing positional relationships between a physical object and a specified body part of a user. The problem addressed is the need for accurate and reliable detection of spatial interactions between objects and human body parts, which is critical in applications such as augmented reality, ergonomics, and human-computer interaction. The method involves capturing an image that includes both the physical object and the specified body part, such as a hand or limb. Image processing techniques are then applied to analyze the captured image and determine whether the positional relationship between the object and the body part meets predetermined criteria. These criteria may include proximity, alignment, or relative orientation, depending on the application. The image processing may involve object recognition, feature extraction, or pose estimation to assess the spatial relationship accurately. The method ensures that the positional relationship is evaluated in real-time or near-real-time, enabling dynamic feedback or adjustments based on the detected interaction. This approach enhances the precision and reliability of human-object interaction analysis, making it suitable for applications requiring precise spatial awareness, such as virtual reality environments, assistive technologies, or industrial safety systems. The invention improves upon existing solutions by leveraging advanced image processing to provide more accurate and context-aware positional assessments.
16. The computer-implemented method according to claim 15 , wherein the wearable display device includes a camera device that captures the image; and wherein the displaying displays the virtual target object and a virtual object other than the virtual target object superimposing with the image obtained by the camera device.
This invention relates to augmented reality (AR) systems, specifically methods for displaying virtual objects in a wearable display device. The problem addressed is the need to enhance user interaction with virtual objects in AR environments by providing a more immersive and contextually relevant display. The method involves a wearable display device equipped with a camera that captures real-world images. The system processes these images to identify a virtual target object and at least one additional virtual object. The virtual target object and the additional virtual object are then superimposed onto the captured image and displayed to the user. This allows the user to see both the virtual target object and other virtual objects in the same field of view, enhancing spatial awareness and interaction within the AR environment. The system may also adjust the display based on user input or environmental factors to improve the user experience. The method ensures that virtual objects are accurately positioned and rendered in real-time, providing a seamless integration between the virtual and physical worlds. This approach is particularly useful in applications such as gaming, navigation, and industrial training, where precise and dynamic AR overlays are required.
17. The computer-implemented method according to claim 14 , wherein the determining determines whether the positional relationship satisfies the predetermined criteria by using a sensor that is attached to at least one of the physical object and the specified body part and that detects proximity of the specified body part to the physical object.
This invention relates to a computer-implemented method for assessing the positional relationship between a physical object and a specified body part of a user. The method addresses the challenge of accurately detecting and evaluating the spatial proximity between a user's body part and an object in real-time, which is critical for applications such as ergonomics, safety monitoring, or interactive systems. The method involves using a sensor attached to either the physical object or the specified body part to detect proximity. The sensor measures the distance or relative position between the body part and the object, then determines whether this positional relationship meets predetermined criteria. These criteria could include thresholds for safe interaction distances, optimal ergonomic positioning, or specific spatial configurations required for a task. The sensor data is processed to generate a binary or quantitative output indicating whether the criteria are satisfied, enabling automated feedback or system adjustments. The method may also involve additional steps such as calibrating the sensor, filtering noise from the sensor data, or integrating multiple sensor inputs for improved accuracy. The system can be applied in various contexts, including industrial safety, medical rehabilitation, or virtual reality interactions, where precise positional tracking is essential. The use of a dedicated sensor ensures reliable and real-time assessment, enhancing user safety and system performance.
18. The computer-implemented method according to claim 12 , wherein displaying, on the screen, the virtual target object with the display position and display arrangement determined based on the second three-dimensional position and the second arrangement reduces a difference between visual perception and tactile perception of a user viewing the screen while the physical object is being manipulated by the specified body part.
This invention relates to a computer-implemented method for enhancing user interaction with virtual and physical objects in an augmented reality (AR) or virtual reality (VR) environment. The method addresses the problem of misalignment between visual and tactile feedback when a user manipulates a physical object while viewing a corresponding virtual object on a screen. This misalignment can cause discomfort, confusion, or reduced accuracy in tasks requiring precise coordination between visual perception and physical manipulation. The method involves tracking the three-dimensional position and arrangement of a physical object being manipulated by a user's body part, such as a hand or finger. A virtual target object is then displayed on a screen, with its position and arrangement adjusted based on the tracked physical object's position and arrangement. This adjustment minimizes the difference between the user's visual perception of the virtual object and their tactile perception of the physical object, improving spatial alignment and reducing cognitive dissonance. The method may also involve dynamically updating the virtual object's display in real-time as the physical object is moved or rotated, ensuring continuous synchronization between the two. The technique is particularly useful in applications where users interact with physical objects while viewing augmented or virtual representations, such as in training simulations, medical procedures, or industrial design. By aligning visual and tactile feedback, the method enhances user experience, accuracy, and efficiency in such environments.
19. A virtual reality system comprising: a display control device configured to control display of a virtual reality (VR) space; a physical object arranged within a real space and having a corresponding virtual object displayed within the VR space; a VR tracking device configured to provide tracking data of objects within the real space to the display control device; and a camera attached to a wearable display device configured to obtain at least one image of the real space, wherein the display control device performs a process comprising: receiving tracking data from the VR tracking device; estimating position and arrangement of the wearable display device based on the acquired tracking data; estimating position of a tip of a specified object to be used to manipulate the target object based on the at least one image from the camera; detecting an augmented reality (AR) marker within the real space used for estimating position of the target object within the real space; estimating position and arrangement of the target object based on the detection of the AR marker; determining a distance between the estimated position of the tip of the specified object and the estimated position and arrangement of the target object; displaying the VR space including the corresponding virtual object on a display of the wearable display device based on the estimated position and arrangement of the target object estimated using the acquired tracking data when the determined distance is greater than a threshold value; and displaying the VR space including the corresponding virtual object on the display of the wearable display device based on the estimated position and arrangement of the target object estimated using the detection of the AR marker when the determined distance is smaller than a threshold value, wherein the displaying the VR space including the corresponding virtual object on the display of the wearable display device based on the estimated position and arrangement of the target object estimated using the detection of the AR marker reduces a difference between visual perception and tactile perception within a mixed reality system when the target object is being physically manipulated while the VR space is displayed.
A virtual reality (VR) system integrates real and virtual environments to enhance user interaction with physical objects. The system includes a display control device managing VR space visualization, a physical object in the real space with a corresponding virtual object in the VR space, and a VR tracking device providing real-space object tracking data. A camera on a wearable display captures real-space images to estimate the position of a specified object's tip used for manipulating a target object. An augmented reality (AR) marker in the real space aids in determining the target object's position and orientation. The system calculates the distance between the specified object's tip and the target object. If the distance exceeds a threshold, the VR space displays the virtual object based on tracking data. If the distance is below the threshold, the system uses AR marker data to adjust the virtual object's position, reducing discrepancies between visual and tactile feedback during physical manipulation. This approach improves mixed reality accuracy by dynamically switching between tracking methods to align virtual and real-world interactions seamlessly.
Unknown
February 25, 2020
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.