Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A computer-implemented method comprising: at least one processing device; and memory storing instructions that when executed cause the processing device to perform operations including: obtaining, from a first tracking system, an initial three-dimensional (3D) position of an electronic device in relation to image features captured by a camera of the electronic device; obtaining, from a second tracking system, an orientation associated with the electronic device; and responsive to detecting a movement of the electronic device: obtaining, from the second tracking system, an updated orientation associated with the detected movement of the electronic device; generating and providing a query to the first tracking system, the query corresponding to at least a portion of the image features and including the updated orientation and the initial 3D position of the electronic device; receiving, responsive to the query, a plurality of position changes for the portion of the image features in relation to the initial 3D position of the electronic device; generating, for a sampled number of the plurality of position changes, an updated 3D position for the electronic device; generating a 6-DoF pose using the updated 3D positions and the updated orientation for the electronic device; and providing, for display on the electronic device, a camera feed depicting movement of the image features based on the movement of the electronic device, according to the generated 6-DoF pose.
2. The method of claim 1 , wherein the updated 3D positions are generated using a periodic sampling of three dimensions of data for a plurality of image frames representing the position of the portion of the image features relative to the position of the electronic device.
3. The method of claim 2 , wherein the periodic sampling is performed using a threshold frame rate configured to reduce jitter in the movement of the portion of the image features depicted in the camera feed provided based on the generated 6-DoF pose.
4. The method of claim 1 , wherein providing the camera feed depicting movement of the image features based on the movement of the electronic device according to the 6-DoF pose includes providing placement of virtual objects associated with the user in the camera feed according to the 6-DoF pose each time the electronic device is moved.
5. The method of claim 1 , wherein the image features include: portions of a face of a user being captured by the camera of the electronic device, the camera being a front facing camera; and augmented reality content associated with the user being captured by the front facing camera.
6. The method of claim 1 , wherein: the first tracking system executes a facial feature tracking algorithm configured to determine 3D location changes for the image features associated with at least one selected facial feature; and the second tracking system is an inertial measurement unit (IMU) installed on the electronic device.
7. The method of claim 1 , wherein combining output from the first tracking system and output from the second tracking system enables tracking and placement of augmented reality content based on the generated 6-DoF pose, and responsive to the detected movement of the electronic device.
8. The method of claim 1 , wherein obtaining the updated orientation associated with the detected movement of the electronic device from the second tracking system is performed in response to determining that the first tracking system is unable to provide both the position and orientation with 6-DoF.
9. An electronic device comprising: a first tracking system configured to generate a 6-DoF pose for the electronic device corresponding to image features depicted in a camera feed displayed by the electronic device, the 6-DoF pose being generated from: a determined orientation for the electronic device, and a determined position for the electronic device, the determined position calculated using a facial feature tracking algorithm configured to detect three-dimensional location changes for at least one selected facial feature in the image features in the camera feed displayed by the electronic device; a second tracking system including at least one inertial measurement unit (IMU) for determining an orientation of the electronic device in three-dimensional space; and at least one processor coupled to memory and configured to: trigger the first tracking system to generate the 6-Dof pose for the electronic device if the first tracking system operates within a predefined confidence threshold; trigger the second tracking system to generate an alternate 6-DoF pose if the first tracking system failed to operate within the predefined confidence threshold, the alternate 6-DoF pose generated by combining the determined position from the first tracking system and the orientation of the second tracking system; and trigger, for display on the electronic device, an updated camera feed depicting movement of the image features based on the 6-DoF pose or the alternate 6-DoF pose according to the determined operation of the first tracking system with respect to the predefined confidence threshold.
10. The electronic device of claim 9 , wherein the determination of whether the first tracking system operates within the predefined confidence threshold is performed upon detecting movement of the electronic device.
11. The electronic device of claim 9 , wherein the facial feature tracking algorithm of the first tracking system is configured to perform, upon detecting movement of the electronic device, a determination of an updated position of the electronic device relative to the at least one facial feature, the determination of the updated position of the electronic device including performing periodic sampling of three dimensions of data of a plurality of images of the at least one facial feature to reduce jitter in the movement of the at least one facial feature upon triggering the updated camera feed for display on the electronic device.
This invention relates to electronic devices with facial feature tracking systems for stabilizing camera feeds. The problem addressed is jitter or instability in displayed camera feeds when the device moves, particularly during facial tracking. The solution involves a tracking system that detects device movement and adjusts the camera feed by periodically sampling three-dimensional data from multiple images of facial features. This sampling reduces jitter in the displayed movement of the tracked features. The system uses a facial feature tracking algorithm that processes images to determine the device's updated position relative to the tracked features. By analyzing multiple images and sampling data in three dimensions, the algorithm compensates for movement, ensuring smoother display of the tracked features. The invention improves user experience by minimizing visual disturbances caused by device motion during facial tracking applications, such as augmented reality or video calls. The tracking system may operate in conjunction with other components, such as a camera and display, to provide real-time adjustments to the camera feed based on the tracked facial features.
12. The electronic device of claim 9 , further comprising at least one communication module to trigger transmission of the 6-DoF pose or the alternate 6-DoF pose to display the image features on the electronic device based on a plurality of detected movements of the electronic device.
This invention relates to electronic devices equipped with motion tracking and display capabilities, particularly for rendering image features based on device movements. The device includes a sensor system to detect movements and determine a 6-DoF (six degrees of freedom) pose, which defines the device's position and orientation in 3D space. If the sensor system fails to accurately determine the 6-DoF pose, an alternate 6-DoF pose is computed using a secondary method, such as sensor fusion or predictive algorithms. The device further includes a communication module that transmits the 6-DoF pose or the alternate 6-DoF pose to a display system. The display system uses this pose data to render and display image features, such as virtual objects or augmented reality elements, in alignment with the device's movements. The communication module triggers this transmission based on a plurality of detected movements, ensuring real-time synchronization between the device's physical motion and the displayed content. This system enhances user interaction by dynamically adjusting visual output in response to device movements, improving immersion in applications like augmented reality, virtual reality, or motion-based gaming. The invention addresses challenges in maintaining accurate pose tracking and seamless visual feedback in dynamic environments.
13. The electronic device of claim 9 , wherein the 6-DoF pose and the alternate 6-DoF pose indicate a position of the electronic device relative to the at least one selected facial feature.
14. A computer program product tangibly embodied on a non-transitory computer-readable medium and comprising instructions that, when executed, are configured to cause at least one processor to: obtain, from a first tracking system, an initial three-dimensional (3D) position of an electronic device in relation to image features captured by a camera of the electronic device; obtain, from a second tracking system, an orientation associated with the electronic device; and responsive to detecting a movement of the electronic device: obtain, from the second tracking system, an updated orientation associated with the detected movement of the electronic device; generate and provide a query to the first tracking system, the query corresponding to at least a portion of the image features and including the updated orientation and the initial 3D position of the electronic device; receive, responsive to the query, a plurality of position changes for the portion of the image features in relation to the initial 3D position of the electronic device; generate, for a sampled number of the plurality of position changes, an updated 3D position for the electronic device; generate a 6-DoF pose using the updated 3D positions and the updated orientation for the electronic device; and provide, for display on the electronic device, a camera feed depicting movement of the image features based on the movement of the electronic device, according to the generated 6-DoF pose.
15. The computer program product of claim 14 , wherein the updated 3D positions are generated using a periodic sampling of three dimensions of data for a plurality of image frames representing the position of the portion of the image features relative to the position of the electronic device.
16. The computer program product of claim 14 , wherein providing the camera feed depicting movement of the image features based on the movement of the electronic device according to the 6-DoF pose includes providing placement of virtual objects associated with the user in the camera feed according to the 6-DoF pose each time the electronic device is moved.
This invention relates to augmented reality (AR) systems that integrate virtual objects into a live camera feed based on the movement of an electronic device. The technology addresses the challenge of accurately aligning virtual objects with real-world environments in real-time, ensuring they appear stable and properly positioned as the device moves. The system tracks the device's six degrees of freedom (6-DoF) pose—translation in three axes (X, Y, Z) and rotation around three axes (roll, pitch, yaw)—to dynamically adjust the placement of virtual objects in the camera feed. This ensures that when the device moves, the virtual objects move correspondingly, maintaining spatial coherence with the real-world scene. The invention also includes methods for capturing and processing image features from the camera feed to determine the device's pose and updating the virtual object positions accordingly. The system may involve real-time tracking algorithms, sensor fusion (e.g., combining camera data with inertial measurement unit (IMU) data), and rendering techniques to overlay virtual objects seamlessly. The goal is to enhance AR experiences by providing stable, contextually accurate virtual object placement that responds to user movement.
17. The computer program product of claim 14 , wherein the image features include: portions of a face of a user being captured by the camera of the electronic device, the camera being a front facing camera; and augmented reality content associated with the user being captured by the front facing camera.
18. The computer program product of claim 14 , wherein: the first tracking system executes a facial feature tracking algorithm configured to determine 3D location changes for the image features associated with at least one selected facial feature; and the second tracking system is an inertial measurement unit (IMU) installed on the electronic device.
19. The computer program product of claim 14 , wherein combining output from the first tracking system and output from the second tracking system enables tracking and placement of augmented reality content based on the generated 6-DoF pose, and responsive to the detected movement of the electronic device.
20. The computer program product of claim 14 , wherein obtaining the updated orientation associated with the detected movement of the electronic device from the second tracking system is performed in response to determining that the first tracking system is unable to provide both the position and orientation with 6-DoF.
This invention relates to a computer program product for tracking the position and orientation of an electronic device using multiple tracking systems. The problem addressed is the need for accurate and reliable 6-DoF (six degrees of freedom) tracking, which includes both position and orientation, in scenarios where a primary tracking system may fail or degrade in performance. The solution involves a system that dynamically switches between or combines data from multiple tracking systems to ensure continuous and precise tracking. The invention includes a first tracking system capable of providing 6-DoF data and a second tracking system that may provide orientation data when the first system is unable to deliver both position and orientation. The computer program product monitors the performance of the first tracking system and, if it detects that the first system cannot provide both position and orientation with 6-DoF accuracy, it obtains updated orientation data from the second tracking system. This ensures that the device's orientation remains accurately tracked even if the primary system experiences limitations. The system may also integrate data from both tracking systems to improve overall accuracy and reliability. The invention is particularly useful in applications requiring high-precision tracking, such as augmented reality, virtual reality, robotics, and navigation systems.
Unknown
February 9, 2021
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.