Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A method for characterizing a gaze position of a user in a query image, the method comprising: obtaining a query image of a user captured by a camera of a mobile device; obtaining device characteristics data specifying (i) characteristics of the mobile device, (ii) characteristics of the camera of the mobile device, or (iii) both; maintaining data associating the device characteristics data with current values for a plurality of device-dependent parameters of a gaze prediction neural network; and processing a neural network input comprising one or more images derived from the query image using the gaze prediction neural network, wherein: the gaze prediction neural network has a plurality of parameters comprising (i) a plurality of device-independent parameters and (ii) the plurality of device-dependent parameters, wherein device-independent parameters are parameters that, after training, have the same values regardless of the device characteristics of the mobile device, and device-dependent parameters are parameters that have different values for different device characteristics of the mobile device; the gaze prediction neural network is configured to, at run time and after the gaze prediction neural network has been trained, process the neural network input to generate a neural network output that characterizes a gaze position of the user in the query image; and processing the neural network input comprises setting the values of the device-dependent parameters to the current values associated with the device characteristics data.
This invention relates to a method for determining a user's gaze position in an image captured by a mobile device. The problem addressed is the variability in gaze estimation accuracy across different mobile devices due to differences in hardware characteristics, such as camera specifications and device configurations. The solution involves a gaze prediction neural network that adapts to device-specific characteristics while maintaining a core set of device-independent parameters. The method begins by capturing a query image of the user using the mobile device's camera. Device characteristics data, which may include hardware specifications of the mobile device or its camera, is obtained. This data is used to retrieve pre-trained values for device-dependent parameters of the neural network, which are adjusted based on the specific device characteristics. The neural network processes the query image or derived images, combining both device-independent and device-dependent parameters to predict the user's gaze position. The device-independent parameters remain consistent across devices, while the device-dependent parameters are dynamically adjusted to account for variations in hardware, improving accuracy without requiring retraining for each device. This approach enables robust gaze estimation across diverse mobile devices.
2. The method of claim 1 , wherein the obtaining the image, the obtaining the characteristics data, and the processing the neural network input are performed by the mobile device.
A mobile device captures an image of a physical object and extracts characteristics data from the image. The device processes this data to generate a neural network input, which is then used to analyze the object. The neural network is trained to recognize specific features or patterns in the input data, enabling tasks such as object identification, classification, or measurement. The entire process, from image capture to neural network processing, is performed locally on the mobile device, eliminating the need for external servers or cloud-based processing. This approach enhances privacy, reduces latency, and ensures functionality in environments with limited or no network connectivity. The method leverages the computational capabilities of modern mobile devices to perform complex neural network operations efficiently. The characteristics data may include geometric features, texture information, or other relevant attributes extracted from the image. The neural network input is structured to optimize recognition accuracy while minimizing computational overhead. This solution is particularly useful in applications requiring real-time analysis, such as augmented reality, industrial inspection, or medical diagnostics, where immediate feedback and data privacy are critical.
3. The method of claim 1 , wherein the gaze prediction neural network comprises a plurality of neural network layers configured to apply the device-independent-parameters to generate an initial neural network output that characterizes an initial predicted gaze position of the user, and wherein the gaze prediction neural network is configured to adjust the initial neural network output in accordance with at least some of the device-dependent parameters to generate the neural network output.
This invention relates to gaze prediction systems that use neural networks to estimate a user's gaze position. The core problem addressed is improving the accuracy of gaze prediction across different devices by separating device-independent and device-dependent parameters. The system employs a neural network with multiple layers that first processes device-independent parameters to generate an initial predicted gaze position. These parameters may include general user-specific or environmental factors that influence gaze behavior. The neural network then refines this initial prediction by incorporating device-dependent parameters, such as hardware-specific calibration data or sensor characteristics, to produce a final, more accurate gaze prediction. This two-stage approach allows the system to adapt to variations between different devices while maintaining a consistent baseline prediction model. The method ensures that the neural network can generalize across devices while fine-tuning predictions based on specific device characteristics, improving overall accuracy and reliability in gaze tracking applications.
4. The method of claim 3 , wherein adjusting the initial neural network output comprises: applying a linear device dependent parameters transformation to the initial neural network output.
A system and method for improving neural network performance by adjusting initial outputs through device-dependent parameter transformations. The technology addresses the challenge of optimizing neural network predictions across different hardware devices, where variations in computational precision, memory constraints, or processing capabilities can degrade accuracy. The method involves generating an initial output from a neural network and then applying a linear transformation to this output. The transformation is tailored to the specific characteristics of the target device, such as its computational limitations or hardware-specific biases. This adjustment compensates for device-specific deviations, ensuring consistent and reliable performance across diverse hardware platforms. The transformation may include scaling, shifting, or other linear operations to correct systematic errors introduced by the device. By dynamically adapting the neural network's output to the device's operational constraints, the method enhances accuracy and robustness without requiring extensive retraining or hardware-specific model modifications. This approach is particularly useful in edge computing, embedded systems, and other environments where hardware variability impacts neural network deployment.
5. The method of claim 1 , wherein the neural network input further comprises data specifying a location of one or more eye landmarks in the query image.
A system and method for analyzing images using a neural network to detect and classify objects, with an enhanced input that includes eye landmark data. The technology operates in the domain of computer vision and image processing, addressing the challenge of improving object detection accuracy by incorporating additional facial feature information. The neural network processes a query image to identify objects, and the input to the network includes data specifying the locations of one or more eye landmarks within the image. These eye landmarks serve as additional spatial references, helping the neural network better understand facial structure and orientation, which can improve detection of facial features, expressions, or other objects in the image. The system may also include preprocessing steps to extract and normalize the eye landmark data before feeding it into the neural network. This approach enhances the network's ability to handle variations in lighting, pose, and occlusion, leading to more robust and accurate object detection. The method is particularly useful in applications such as facial recognition, emotion analysis, and augmented reality, where precise facial feature detection is critical.
6. The method of claim 5 , wherein the gaze prediction neural network is configured to: apply at least some of the device-dependent parameters to adjust the location of the eye landmarks; and process the adjusted location of the eye landmarks and the one or more images in accordance with the device-independent parameters to generate an initial neural network output.
This invention relates to gaze prediction systems that use neural networks to estimate eye gaze direction from images of a user's eyes. The problem addressed is the variability in eye landmark detection across different devices, which can reduce the accuracy of gaze prediction. The solution involves a neural network that adjusts eye landmark locations based on device-dependent parameters before processing them with device-independent parameters to generate a gaze prediction. The system first captures one or more images of a user's eyes using a camera. Eye landmarks, such as pupil centers or corneal reflections, are detected in these images. Device-dependent parameters, which account for variations in camera hardware or calibration, are applied to adjust the detected landmark locations. These adjusted landmarks are then processed alongside the original images using device-independent parameters, which generalize across different devices. The neural network combines these inputs to generate an initial gaze prediction output. By separating device-specific adjustments from the core gaze prediction logic, the system improves accuracy across different devices without requiring extensive recalibration. The neural network's architecture allows it to learn both the necessary adjustments for specific hardware and the general patterns of eye movement. This approach enhances robustness in real-world applications where users may interact with multiple devices.
7. The method of claim 1 , further comprising: obtaining one or more calibration images of the user captured using the camera of the mobile device and, for each of the calibration images, a respective calibration label that labels a known gaze position of the user in the calibration image; and training the gaze prediction neural network using the one or more calibration images to determine the current values for the set of device-dependent parameters from initial values for the set of device-dependent parameters while holding the device-independent parameters fixed.
This invention relates to gaze tracking using a mobile device camera, addressing the challenge of accurately predicting a user's gaze position despite variations in device hardware and environmental conditions. The method involves capturing one or more calibration images of the user with the mobile device camera, where each calibration image is paired with a corresponding calibration label indicating the known gaze position in that image. A gaze prediction neural network is trained using these calibration images to adjust device-dependent parameters while keeping device-independent parameters fixed. The device-dependent parameters account for variations in camera hardware, such as lens distortion or sensor characteristics, while the device-independent parameters represent general gaze prediction models applicable across different devices. By calibrating the network with user-specific data, the system improves accuracy in real-time gaze tracking, compensating for individual differences in facial features and lighting conditions. The trained network then predicts the user's gaze position in subsequent images captured by the mobile device camera, enabling applications like eye-tracking for user interface interaction or attention monitoring. The calibration process ensures robustness across diverse mobile devices without requiring extensive pre-training on each specific hardware configuration.
8. The method of claim 1 , wherein the one or more images derived from the query image comprise a respective image crop corresponding to each of one or more eyes of the user.
This invention relates to image processing techniques for analyzing user gaze or eye-related features from a query image. The method involves extracting one or more images from the query image, where each extracted image is a cropped region corresponding to a user's eye. These eye crops are then processed to derive gaze direction, eye movement, or other ocular metrics. The technique may be used in applications such as gaze tracking, user authentication, or attention monitoring. The method ensures precise alignment of the cropped regions with the user's eyes, improving accuracy in subsequent analysis. The system may include preprocessing steps like noise reduction or contrast enhancement to optimize the quality of the extracted eye crops. The invention addresses challenges in accurately isolating eye regions from complex backgrounds or varying lighting conditions, ensuring reliable eye tracking in real-world scenarios. The extracted eye crops can be further analyzed using machine learning models or computer vision algorithms to determine gaze direction, blink patterns, or other biometric data. This approach enhances the robustness of eye-tracking systems in applications like virtual reality, medical diagnostics, or human-computer interaction.
9. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising: obtaining a query image of a user captured by a camera of a mobile device; obtaining device characteristics data specifying (i) characteristics of the mobile device, (ii) characteristics of the camera of the mobile device, or (iii) both; maintaining data associating the device characteristics data with current values for a plurality of device-dependent parameters of a gaze prediction neural network; and processing a neural network input comprising one or more images derived from the query image using the gaze prediction neural network, wherein: the gaze prediction neural network has a plurality of parameters comprising (i) a plurality of device-independent parameters and (ii) the plurality of device-dependent parameters, wherein device-independent parameters are parameters that, after training, have the same values regardless of the device characteristics of the mobile device, and device-dependent parameters are parameters that have different values for different device characteristics of the mobile device; the gaze prediction neural network is configured to, at run time and after the gaze prediction neural network has been trained, process the neural network input to generate a neural network output that characterizes a gaze position of the user in the query image; and processing the neural network input comprises setting the values of the device-dependent parameters to the current values associated with the device characteristics data.
The system involves a gaze prediction neural network that estimates a user's gaze position from an image captured by a mobile device camera. The challenge addressed is the variability in gaze prediction accuracy across different mobile devices due to differences in hardware characteristics. The solution uses a neural network with two types of parameters: device-independent parameters, which remain constant regardless of the device, and device-dependent parameters, which are adjusted based on the specific characteristics of the mobile device. The system obtains a query image of the user and device characteristics data, which may include details about the mobile device and its camera. The system maintains a mapping between device characteristics and corresponding values for the device-dependent parameters. During processing, the neural network input, derived from the query image, is processed using the neural network, where the device-dependent parameters are set to values associated with the device characteristics data. This approach allows the neural network to adapt to different devices while maintaining consistent performance. The output is a characterization of the user's gaze position in the query image.
10. The system of claim 9 , wherein the obtaining the image, the obtaining the characteristics data, and the processing the neural network input are performed by the mobile device.
A mobile device system captures an image of a physical object and extracts characteristics data from the image. The system processes the image and characteristics data to generate a neural network input, which is then analyzed by a neural network to determine a classification or other output related to the object. The neural network may be trained to recognize specific features, patterns, or attributes of the object based on the extracted characteristics. The mobile device performs all steps, including image capture, data extraction, and neural network processing, locally without requiring external servers or additional devices. This enables real-time analysis and classification of objects using only the mobile device's hardware and software capabilities. The system may be used for applications such as object recognition, quality inspection, or augmented reality, where immediate feedback and processing efficiency are important. The neural network may be pre-trained or updated based on new data collected by the mobile device to improve accuracy over time. The system ensures privacy and reduces latency by performing all computations on the device itself.
11. The system of claim 9 , wherein the gaze prediction neural network comprises a plurality of neural network layers configured to apply the device-independent-parameters to generate an initial neural network output that characterizes an initial predicted gaze position of the user, and wherein the gaze prediction neural network is configured to adjust the initial neural network output in accordance with at least some of the device-dependent parameters to generate the neural network output.
A gaze prediction system uses a neural network to estimate a user's gaze position based on input data. The system addresses challenges in accurately predicting gaze across different devices by separating device-independent and device-dependent parameters. The neural network processes device-independent parameters through multiple layers to generate an initial predicted gaze position. This initial output is then refined using device-dependent parameters, which account for variations between different devices, such as camera placement or sensor characteristics. The refined output provides an accurate gaze prediction tailored to the specific device being used. This approach improves gaze tracking consistency across diverse hardware configurations, enhancing applications like virtual reality, eye-tracking interfaces, and assistive technologies. The system leverages deep learning to adapt predictions dynamically, ensuring robustness in real-world scenarios where device-specific factors can significantly impact accuracy. By decoupling device-independent and device-dependent processing, the system achieves both generalizability and precision.
12. The system of claim 11 , wherein adjusting the initial neural network output comprises: applying a linear device dependent parameters transformation to the initial neural network output.
A system for processing neural network outputs includes a neural network configured to generate an initial output based on input data. The system further includes a transformation module that adjusts the initial neural network output by applying a linear transformation using device-dependent parameters. These parameters are specific to the hardware or software environment in which the neural network operates, ensuring the output is optimized for the particular device or system. The transformation accounts for variations in processing capabilities, memory constraints, or other device-specific factors that may affect neural network performance. By applying this linear transformation, the system improves the accuracy, efficiency, or compatibility of the neural network output across different devices or platforms. The transformation may involve scaling, shifting, or other linear operations tailored to the device's characteristics. This approach enhances the adaptability of neural networks in diverse computing environments, addressing challenges related to hardware variability and ensuring consistent performance.
13. The system of claim 9 , wherein the neural network input further comprises data specifying a location of one or more eye landmarks in the query image.
A system for analyzing facial images using a neural network processes a query image to detect and classify facial features. The system includes a neural network trained to receive input data representing the query image and additional data specifying the locations of one or more eye landmarks within the image. The neural network generates an output that identifies or classifies facial features based on this combined input. The eye landmark data may include coordinates or positional information indicating the locations of key points around the eyes, such as corners, centers, or other distinctive features. This additional input helps the neural network improve accuracy in detecting and classifying facial features by providing spatial context. The system may be used in applications such as facial recognition, emotion detection, or gaze tracking, where precise feature localization is critical. The neural network may be a convolutional neural network (CNN) or another type of deep learning model optimized for image analysis. The system may also include preprocessing steps to extract or enhance the eye landmarks before feeding them into the neural network. The inclusion of eye landmark data helps the neural network focus on relevant regions of the face, reducing ambiguity and improving performance in tasks requiring detailed facial analysis.
14. The system of claim 13 , wherein the gaze prediction neural network is configured to: apply at least some of the device-dependent parameters to adjust the location of the eye landmarks; and process the adjusted location of the eye landmarks and the one or more images in accordance with the device-independent parameters to generate an initial neural network output.
This invention relates to gaze tracking systems that predict a user's gaze direction using neural networks. The problem addressed is the variability in gaze tracking accuracy across different devices due to differences in camera placement, resolution, and other hardware-specific factors. The system uses a neural network trained with device-dependent parameters to adjust the detected positions of eye landmarks (such as pupils, irises, or corners of the eyes) to account for device-specific variations. After adjusting these landmarks, the system processes the modified landmark positions along with one or more images of the user's eyes using device-independent parameters to generate an initial gaze prediction. This approach improves accuracy by normalizing device-specific differences while maintaining a consistent prediction model. The system may also include preprocessing steps to enhance the input images, such as noise reduction or contrast adjustment, before feeding them into the neural network. The neural network may be a convolutional neural network (CNN) or another type of deep learning model optimized for gaze estimation. The final output is a predicted gaze direction, which can be used in applications like virtual reality, augmented reality, or human-computer interaction.
15. The system of claim 9 , the operations further comprising: obtaining one or more calibration images of the user captured using the camera of the mobile device and, for each of the calibration images, a respective calibration label that labels a known gaze position of the user in the calibration image; and training the gaze prediction neural network using the one or more calibration images to determine the current values for the set of device-dependent parameters from initial values for the set of device-dependent parameters while holding the device-independent parameters fixed.
A system for gaze tracking on mobile devices addresses the challenge of accurately predicting a user's gaze position using device-specific hardware, such as cameras and sensors, which can vary significantly across different mobile devices. The system employs a neural network trained to predict gaze position based on input data from the device's camera and other sensors. The neural network includes both device-independent parameters, which generalize across different devices, and device-dependent parameters, which are specific to the hardware of a particular mobile device. To adapt the system to a specific device, the system obtains one or more calibration images of the user captured by the device's camera, along with corresponding calibration labels that indicate the known gaze position in each image. The system then trains the gaze prediction neural network using these calibration images to adjust the device-dependent parameters from their initial values while keeping the device-independent parameters fixed. This calibration process ensures that the neural network accurately predicts gaze position for the specific hardware configuration of the mobile device, improving accuracy and reliability across different devices. The system thus enables personalized gaze tracking without requiring extensive retraining of the entire neural network.
16. The system of claim 9 , wherein the one or more images derived from the query image comprise a respective image crop corresponding to each of one or more eyes of the user.
The invention relates to a system for processing facial images, specifically focusing on eye regions for applications such as gaze tracking, authentication, or user interaction. The system captures or receives a query image of a user's face and derives one or more images from it, each containing a cropped region corresponding to one or more of the user's eyes. These eye crops are extracted from the query image to isolate the eye regions for further analysis. The system may use these cropped images to determine gaze direction, verify identity, or enable eye-based input. The eye crops are generated by identifying the eye locations within the query image and then extracting the relevant portions, ensuring that each eye is individually isolated for precise processing. This approach improves accuracy in eye-related applications by reducing interference from other facial features and background elements. The system may also include additional processing steps, such as normalization or enhancement, to optimize the eye crops for subsequent analysis. The invention is particularly useful in scenarios requiring high-precision eye tracking or authentication, such as augmented reality, security systems, or medical diagnostics.
17. One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: obtaining a query image of a user captured by a camera of a mobile device; obtaining device characteristics data specifying (i) characteristics of the mobile device, (ii) characteristics of the camera of the mobile device, or (iii) both; maintaining data associating the device characteristics data with current values for a plurality of device-dependent parameters of a gaze prediction neural network; and processing a neural network input comprising one or more images derived from the query image using the gaze prediction neural network, wherein: the gaze prediction neural network has a plurality of parameters comprising (i) a plurality of device-independent parameters and (ii) the plurality of device-dependent parameters, wherein device-independent parameters are parameters that, after training, have the same values regardless of the device characteristics of the mobile device, and device-dependent parameters are parameters that have different values for different device characteristics of the mobile device; the gaze prediction neural network is configured to, at run time and after the gaze prediction neural network has been trained, process the neural network input to generate a neural network output that characterizes a gaze position of the user in the query image; and processing the neural network input comprises setting the values of the device-dependent parameters to the current values associated with the device characteristics data.
This invention relates to gaze prediction in mobile devices using a neural network that adapts to device-specific characteristics. The problem addressed is the variability in gaze estimation accuracy across different mobile devices due to differences in hardware, such as camera specifications, sensor quality, and device form factors. The solution involves a gaze prediction neural network with a hybrid parameter structure, combining device-independent parameters (fixed across devices) and device-dependent parameters (adjusted per device). The system obtains a query image of a user from a mobile device camera and retrieves device characteristics data, which may include hardware specifications of the mobile device or its camera. The neural network uses this data to dynamically set the values of its device-dependent parameters, ensuring accurate gaze prediction regardless of device variations. The network processes the query image to generate an output characterizing the user's gaze position. This approach improves gaze tracking consistency across diverse mobile devices by accounting for hardware differences without requiring retraining the entire model for each device. The invention enhances applications like eye-tracking interfaces, security authentication, and user behavior analysis on mobile platforms.
18. The non-transitory computer storage media of claim 17 , wherein the obtaining the image, the obtaining the characteristics data, and the processing the neural network input are performed by the mobile device.
A system and method for processing images using a neural network on a mobile device. The technology addresses the challenge of efficiently analyzing images captured by mobile devices, particularly in scenarios where real-time processing or offline functionality is required. The invention involves a mobile device that captures an image and extracts characteristics data from the image, such as features, objects, or other relevant information. This data is then processed as input to a neural network, which performs tasks such as classification, object detection, or segmentation. The neural network may be pre-trained and stored locally on the device, enabling on-device processing without relying on external servers. The system ensures that the entire workflow—image capture, data extraction, and neural network processing—is executed locally on the mobile device, reducing latency and improving privacy by minimizing data transmission to remote servers. This approach is particularly useful for applications requiring immediate feedback, such as augmented reality, medical imaging, or industrial inspections, where connectivity may be unreliable or unavailable. The invention optimizes computational efficiency by leveraging the mobile device's hardware capabilities while maintaining accuracy in neural network predictions.
19. The non-transitory computer storage media of claim 17 , wherein the gaze prediction neural network comprises a plurality of neural network layers configured to apply the device-independent-parameters to generate an initial neural network output that characterizes an initial predicted gaze position of the user, and wherein the gaze prediction neural network is configured to adjust the initial neural network output in accordance with at least some of the device-dependent parameters to generate the neural network output.
This invention relates to gaze prediction systems using neural networks, addressing the challenge of accurately predicting a user's gaze position across different devices with varying hardware characteristics. The system employs a gaze prediction neural network that processes both device-independent parameters (e.g., user-specific eye movement data) and device-dependent parameters (e.g., camera resolution, screen size) to generate a refined gaze prediction. The neural network includes multiple layers that first apply the device-independent parameters to produce an initial predicted gaze position. This initial output is then adjusted using at least some of the device-dependent parameters to account for device-specific variations, resulting in a final neural network output that accurately reflects the user's gaze position regardless of the device being used. The system ensures adaptability across different hardware configurations while maintaining prediction accuracy, making it suitable for applications in human-computer interaction, augmented reality, and accessibility technologies.
20. The non-transitory computer storage media of claim 17 , wherein the operations further comprise: obtaining one or more calibration images of the user captured using the camera of the mobile device and, for each of the calibration images, a respective calibration label that labels a known gaze position of the user in the calibration image; and training the gaze prediction neural network using the one or more calibration images to determine the current values for the set of device-dependent parameters from initial values for the set of device-dependent parameters while holding the device-independent parameters fixed.
This invention relates to gaze tracking systems for mobile devices, specifically improving the accuracy of gaze prediction by adapting to device-specific variations. The problem addressed is the difficulty in accurately predicting a user's gaze position on a mobile device screen due to differences in camera hardware, placement, and environmental factors across devices. Existing gaze tracking models often rely on device-independent parameters, which may not account for these variations, leading to reduced accuracy. The solution involves a gaze prediction neural network that includes both device-independent and device-dependent parameters. The device-independent parameters are pre-trained and fixed, while the device-dependent parameters are calibrated for each specific device. During calibration, the system obtains one or more calibration images of the user captured by the mobile device's camera, along with corresponding calibration labels indicating the known gaze positions in those images. The neural network is then trained using these calibration images to adjust the device-dependent parameters from initial values while keeping the device-independent parameters unchanged. This calibration process ensures that the gaze prediction model adapts to the unique characteristics of the mobile device, improving accuracy without requiring extensive retraining of the entire model. The approach leverages user-specific calibration data to refine the model, making it more reliable across different devices.
Unknown
November 24, 2020
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.