An artificial reality system is described includes a hand-held controller tracking sub-system having two components, a Field-of-View (FOV) tracker and a non-FOV tracker that applies specialized motion models when one or more of controllers are not trackable within the field of view. In particular, under typical operating conditions, the FOV tracker receives state data for a Head Mounted Display (HMD) and controller state data (velocity, acceleration etc.) of a controller to compute estimated poses for the controller. If the controller is trackable (e.g., within the field of view and not occluded), then the pose as computed by the FOV tracker is used and the non-FOV tracker is bypassed. If the controller is not trackable within the field of view and the controller state data meets activation conditions for one or more corner tracking cases, then the non-FOV tracker applies one or more of specialized motion models to compute a controller pose for the controller.
Legal claims defining the scope of protection, as filed with the USPTO.
1. An artificial reality system comprising: an image capture device configured to capture image data representative of a physical environment; a head mounted display (HMD) configured to output artificial reality content; a pose tracker executable by one or more processors and configured to determine, based at least in part on controller state data, a controller pose representing a position and orientation of a hand-held controller, the pose tracker including a non-FOV tracker having a plurality of motion models associated with different motion behaviors for the hand-held controller, each of the plurality of motion models associated with one or more respective activation conditions, wherein the non-FOV tracker is configured to concurrently evaluate two or more motion models of the plurality of motion models and determine the controller state data based, at least in part, on the evaluated motion model of the two or more motion models of the plurality of motion models having a highest priority in response to a determination that the hand-held controller is not trackable within the image data and that the two or more motion models of the plurality of motion models are activated; and a rendering engine configured to render for display at the HMD the artificial reality content and at least one graphical object in accordance with the controller pose.
2. The artificial reality system of claim 1 , wherein the plurality of motion models are configured in a subsumption architecture.
3. The artificial reality system of claim 1 , wherein the pose tracker further comprises an FOV tracker configured to determine image-based controller state data based, at least in part, on the image data; wherein the image-based controller state data is provided as input to the non-FOV tracker; and wherein the non-FOV tracker determines the controller state data based on the image-based controller state data in response to a determination that none of the plurality of motion models are activated.
4. The artificial reality system of claim 1 , wherein the non-FOV tracker is further configured to activate a motion model of the plurality of motion models in response to determining that the one or more respective activation conditions associated with the motion model are satisfied based, at least in part, on controller measurement data indicative of a motion behavior for the hand-held controller.
5. The artificial reality system of claim 1 , wherein the determination that the hand-held controller is not trackable within the image data comprises: a determination that a number of emitters of the hand-held controller detectable in the image data is less than a predetermined or configurable threshold, or a determination that the position of the hand-held controller is unreliable.
6. The artificial reality system of claim 1 , wherein the one or more respective activation conditions comprise an indication that the hand-held controller is not within a field of view of the image capture device and a determination that the hand-held controller is in motion within a threshold distance of the HMD; and wherein the one of the plurality of motion models associated with the one or more respective activation conditions is configured to determine the controller state data in accordance with a pivot of a first position associated with the hand-held controller about a second position associated with a virtual elbow.
7. The artificial reality system of claim 1 , further comprising a second hand-held controller; wherein the hand-held controller is obstructed within a field of view of the image capture device by the second hand-held controller; wherein the one or more respective activation conditions of a motion model of the plurality of motion models comprise a determination that a distance between the hand-held controller and the second hand-held controller is below a threshold value and an acceleration of the hand-held controller is out of synchronization with an acceleration of the second hand-held controller; and wherein the motion model is configured to determine one or more position values of the controller state data as a fixed position with respect to a position of a virtual torso.
8. The artificial reality system of claim 1 , further comprising a second hand-held controller; wherein the hand-held controller is obstructed within a field of view of the image capture device by the second hand-held controller; wherein the one or more respective activation conditions of a motion model of the plurality of motion models comprise a determination that a distance between the hand-held controller and the second hand-held controller is below a threshold value and an acceleration of the hand-held controller is substantially synchronized with an acceleration of the second hand-held controller; and wherein the motion model is configured to determine one or more position values of the controller state data in a fixed position with respect to a second position of the second hand-held controller.
9. The artificial reality system of claim 1 , wherein the hand-held controller further comprises an inertial measurement unit (IMU); wherein the one or more respective activation conditions of a motion model of the plurality of motion models comprise a determination that a most-recent absolute value provided by the IMU is below a first threshold value and a variance in a plurality of buffered values provided by the IMU is below a second threshold value; and wherein the motion model is configured to determine one or more position values of the controller state data to be in a fixed position with respect to a world reference frame.
10. The artificial reality system of claim 1 , wherein the pose tracker is configured to determine from the image data and the controller state data for the hand-held controller that the hand-held controller has reentered a field of view of the image capture device; and wherein the pose tracker is configured to interpolate, over a plurality of display frames, the controller pose based on first controller pose determined while the hand-held controller was out of the field of view and second controller state data determined after the hand-held controller reentered the field of view.
11. The artificial reality system of claim 1 , wherein the image capture device is integrated with the HMD.
12. The artificial reality system of claim 1 , wherein the image data comprises infra-red image data.
13. A method comprising: obtaining, by an image capture device of an artificial reality system including a head mounted display (HMD), image data representative of a physical environment; determining that two or more motion models of a plurality of motion models are activated based on controller measurement data for a hand-held controller satisfying one or more activation conditions associated with each corresponding motion model of the plurality of motion models; concurrently evaluating the two or more activated motion models; determining, by a non-FOV tracker of the artificial reality system, controller state data for the hand-held controller in accordance with a motion model of the two or more activated motion models having a highest priority, wherein the determining the controller state data in accordance with the motion model is in response to determining that the hand-held controller is not trackable within the image data; determining, by the artificial reality system, a controller pose representing a position and orientation of the hand-held controller based, at least in part, on the controller state data; and rendering, by the artificial reality system for display at the HMD, artificial reality content and at least one graphical object in accordance with the controller pose.
14. The method of claim 13 , wherein the plurality of motion models are configured in a subsumption architecture.
15. The method of claim 13 , further comprising: determining, by an FOV tracker of the artificial reality system, image-based controller state data for the hand-held controller based, at least in part, on the image data; providing the image-based controller state data as input to the non-FOV tracker; wherein the non-FOV tracker determines the controller state data based, at least in part, on the image-based controller state data in response to determining that none of the plurality of motion models are activated.
16. The method of claim 13 , wherein the determining that the controller measurement data indicative of the motion behavior for the hand-held controller satisfies the one or more activation conditions associated with the motion model comprises determining that the hand-held controller is not trackable within the image capture data and that the hand-held controller is in motion within a threshold distance of the HMD, the method further comprising evaluating the motion model associated with the one or more respective activation conditions, wherein the motion model is configured to determine the controller state data in accordance with a pivot of a first position of the hand-held controller about a second position of a virtual elbow.
17. The method of claim 13 , wherein the hand-held controller is obstructed within a field of view of the image capture device by a second hand-held controller; wherein the determining that the controller measurement data indicative of the motion behavior for the hand-held controller satisfies the one or more activation conditions associated with the motion model comprises determining that a distance between the hand-held controller and the second hand-held controller is below a threshold value and determining that an acceleration of the hand-held controller is out of synchronization with an acceleration of the second hand-held controller, the method further comprising evaluating the motion model associated with the one or more respective activation conditions, wherein the motion model is configured to determine one or more position values of the controller state data as a fixed position with respect to a position of a virtual torso.
18. The method of claim 13 , wherein the hand-held controller is obstructed within a field of view of the image capture device by a second hand-held controller, wherein determining that the controller measurement data indicative of the motion behavior for the hand-held controller satisfies the one or more activation conditions associated with the motion model comprises determining that a distance between the hand-held controller and the second hand-held controller is below a threshold value and an acceleration of the hand-held controller is synchronized with an acceleration of the second hand-held controller, the method further comprising evaluating the motion model associated with the one or more respective activation conditions, wherein the motion model is configured to determine one or more position values of the controller state data in a fixed position with respect to a second position of the second hand-held controller.
19. The method of claim 13 , wherein determining that the controller measurement data indicative of the motion behavior for the hand-held controller satisfies the one or more activation conditions associated with the motion model comprises determining that a most-recent absolute value provided by an inertial measurement unit (IMU) of the hand-held controller is below a first threshold value and a variance in a plurality of buffered values provided by the IMU is below a second threshold value, and the method further comprising evaluating the motion model associated with the one or more respective activation conditions, wherein the motion model is configured to determine one or more position values of the controller state data in a fixed position in a world reference frame.
20. A non-transitory, computer-readable medium comprising instructions that, when executed, cause one or more processors of an artificial reality system including a head mounted display (HMD) to: obtain image data representative of a physical environment via an image capture device; determine that two or more motion models of a plurality of motion models are activated based on controller measurement data for a hand-held controller satisfying one or more activation conditions associated with each corresponding motion model of the plurality of motion models; concurrently evaluate the two or more activated motion models; determine the controller state data for the hand-held controller in accordance with a motion model of the two or more activated motion models having a highest priority, wherein the determination of the controller state data in accordance with the motion model is in response to a determination that the hand-held controller is not trackable within the image data; determine a controller pose representing a position and orientation of the hand-held controller based, at least in part, on the controller state data; and render for display at the HMD, artificial reality content and at least one graphical object in accordance with the controller pose.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
May 20, 2019
December 1, 2020
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.