A system and a method for performing 2D on-screen user gaze estimation using an input facial image of a user captured using a camera associated to a processing device having a display. The method and system allow automated user calibration through automatic recording of calibration samples each including a calibration facial image of the user and an interaction point corresponding to a point on the display where an occurrence of a user interaction was detected when the corresponding calibration image was captured. The system and method also optimize user-specific parameters using the calibration samples by iteratively minimizing a total difference between the interaction points of a plurality of the calibration samples and corresponding 2D gaze estimation results and convert an estimated 3D gaze direction into a 2D gaze estimation result corresponding to a point on the display, by applying the users-specific parameters.
Legal claims defining the scope of protection, as filed with the USPTO.
5. The system of claim 1, wherein the user-specific parameters are defined as θU={kϕ, bϕ, kθ, bθ, s, yO, zO}, where s is a scaling factor, kϕ, bϕ, kθ, bθ specify a linear transform to the estimated 3D gaze direction in pitch and yaw angle form defined as ĝ3D=(ϕ9, θg) resulting in a user-specific estimated 3D gaze direction defined as ĝU3D=(kϕϕg+bθ, kθθg+bθ), and (0, yO, zO) specify the coordinates of an origin of a head coordinate system (HCS) in which the head of the user is positioned in a corresponding one of the calibration facial images.
10. The system of claim9, wherein the user-specific parameters are defined as θU={s, kϕL, bϕL, kθL, bθL, kϕR, bϕR, kθR, bθR, xO, yO, zO}, where s is a scaling factor, kϕL, bϕL, kθL, bθL and kϕR, bϕR, kθR, bθR each specify a linear transform to the corresponding estimated gaze direction in pitch yaw form defined as ĝL3D=(ϕgL, θgL) and ĝR3D=(ϕgR, θgR) for the left eye and the right eye respectively, resulting in a user-specific 3D gaze direction defined as ĝU3DL=(kϕLϕgL+bϕL, bϕL, kθLθgL+bθL) and ĝU3DR=(kϕRϕgR+bϕR, kθRθgR+bθR) for the left eye and the right eye respectively, and (xO,yO,zO) and (−xO,yO,zO) specify the coordinates of an origin of a head coordinate system (HCS) in which the head of the used is positioned in a corresponding one of the calibration facial image for the left eye and the right eye respectively.
15. The method of claim 11, wherein the user-specific parameters are defined as θU={kϕ, bϕ, kθ, bθ, s, yO, zO}, where s is a scaling factor,kϕ, bϕ, kθ, bθ specify a linear transform to the estimated 3D gaze direction in pitch and yaw angle form defined as ĝ3D=(ϕg, θg) resulting in a user-specific estimated 3D gaze direction defined as ĝU3D=(kϕϕg+bθ, kθθg+bθ), and (0, yO, zO) specify the coordinates of an origin of a head coordinate system (HCS) in which the head of the used is positioned in a corresponding one of the calibration facial images.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
June 23, 2021
April 18, 2023
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.