System, Method, Wearable Display Device and Electronic Device for Eye Tracking

The present application claims priority to Chinese Patent Application No. 2023108184464 filed on Jul. 4, 2023 and entitled “SYSTEMS, METHODS, WEARABLE DISPLAY DEVICES AND ELECTRONIC DEVICES FOR EYE TRACKING”, which is incorporated herein by reference in its entirety.

The example embodiments of the present disclosure generally relate to the field of computers, and in particular, to a system, a wearable display device, a method, an electronic device and a computer-readable storage medium for eye tracking.

With the rise of Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) technologies, the use of wearable display devices will become trendy. Providing wearable display devices with stable eye tracking capabilities and accurate eye tracking data may allow users to move freely and interact naturally with their environment.

In a first aspect of the present disclosure, a system for eye tracking is provided, comprising: at least one light source; at least one metasurface configured to receive light emitted from the at least one light source; at least one detector configured to receive light reflected by an eye of the user from the at least one metasurface and convert the received light into an electrical signal; and a controller configured to control a reflection direction of the light from the at least one light source by the at least one metasurface to enable the reflected light to reach the at least one detector by reflection of the eye of the user.

In a second aspect of the present disclosure, a wearable display device is provided, comprising a system of the first aspect.

In a third aspect of the present disclosure, a method for eye tracking is provided, comprising: controlling at least one light source to emit light; controlling at least one metasurface to reflect the light emitted by the at least one light source to an eye of the user; controlling at least one detector to receive light from the at least one metasurface reflected by the eye of the user; and converting the received light into an electrical signal.

In a fourth aspect of the present disclosure, a method for eye tracking is provided, comprising: controlling, by using a modulation signal, at least one light source to emit light; controlling at least one metasurface to reflect the light emitted by the at least one light source to an eye of a user; controlling, by using the modulation signal, at least one detector to receive light from the at least one metasurface reflected by the eye of the user; and converting the received light into an electrical signal.

In a fifth aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, causing the device to perform the method of the third aspect or the fourth aspect.

In a sixth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium having a computer program stored thereon, the computer program when executed by a processor implementing the method of the third aspect or the fourth aspect.

It would be appreciated that the content described in the section is neither intended to identify the key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.

It would be appreciated that before using the technical solution disclosed in each embodiment of the present disclosure, users should be informed of the type, the scope of use, the use scenario, etc. of the personal information involved in the present disclosure in an appropriate manner and authorized in accordance with relevant laws and regulations.

For example, in response to receiving an active request from a user, a prompt message is sent to the user to explicitly prompt the user that the operation requested operation by the user will need to obtain and use the user's personal information, so that users may select whether to provide personal information to the software or the hardware such as an electronic device, an application, a server or a storage medium that perform the operation of the technical solution of the present disclosure according to the prompt information.

As an optional but non-restrictive implementation, in response to receiving the user's active request, the method of sending prompt information to the user may be, for example, a pop-up window in which prompt information may be presented in text. In addition, the pop-up windows may also contain selection controls for users to choose “agree” or “disagree” to provide personal information to electronic devices.

It would be appreciated that the above notification and acquisition of user authorization process are only schematic and do not limit the implementations of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

It would be appreciated that the data involved in this technical solution (comprising but not limited to the data itself, data acquisition or use) shall comply with the requirements of corresponding laws, regulations, and relevant provisions.

The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it would be appreciated that the present disclosure may be implemented in various forms and should not be interpreted as limited to the embodiments described herein. On the contrary, these embodiments are provided for a more thorough and complete understanding of the present disclosure. It would be appreciated that the accompanying drawings and embodiments of the present disclosure are only for the purpose of illustration and are not intended to limit the scope of protection of the present disclosure.

It should be noted that the titles of any sections/subsections provided in this article are not restrictive. This article describes various types of embodiments throughout, and any type of embodiments may be included within any section/subsection. In addition, the embodiments described in any section/subsection may be combined in any way with any other embodiments described in the same section/subsection and/or different section/subsection.

In the description of the embodiments of the present disclosure, the term “comprising”, and similar terms would be appreciated as open inclusion, that is, “comprising but not limited to”. The term “based on” would be appreciated as “at least partially based on”. The term “one embodiment” or “the embodiment” would be appreciated as “at least one embodiment”. The term “some embodiments” would be appreciated as “at least some embodiments”. Other explicit and implicit definitions may also be included below. The terms “first”, “second”, and/or the like may refer to different or same objects. Other explicit and implicit definitions may also be included below.

As briefly mentioned above, the VR technology, AR technology and MR technology integrate virtual content and real scenes by using a combination of a hardware device and various technical means to provide users with a unique sensory experience. VR is using a computer to simulate a virtual world in a three-dimensional space to provide users with an immersive experience in visual, auditory, tactile, and other aspects. AR enables a real environment and a virtual object to overlay into the same space and coexist in real time. MR is a new visualization environment that integrates the real world and the virtual world, where an object in a physical real-world scene coexists with an object in the virtual world in real time.

Eye tracking refers to tracking of an eye movement by measuring a position of a gaze point of an eye or a movement of the eye relative to a head. In an eye tracking system, an optical switch and a mirror are usually used to change a direction of a light beam from a light source and complete a scanning of a human eye. Eye tracking may be applied in the field of Extended Reality (XR) which includes AR, VR, MR and the like.

Some eye tracking systems use the Micro-Electro-Mechanical System (MEMS) technology to enable deflection scanning of a beam direction by a movement of a scanning mirror. However, a scanning range, a scanning speed, and a scanning mode of a MEMS scanning mirror are limited, particularly in an open-loop system, which cannot provide feedback and correction for output deviations and disturbances, resulting in a decrease in the accuracy of a beam guidance. In addition, the physical size of the scanning mirror, the reliability of movement parts, or the process consistency may all become a restriction factor which applied at a near eye display device.

Embodiments of the present disclosure proposes a system for eye tracking. According to various embodiments of the present disclosure, eye tracking systems include: a light source, a metasurface, a detector, and a controller. The controller controls a reflection direction of light from the at least one light source by the metasurface, to enable the reflected light to reach the detector by reflection of an eye of the user. The detector receives the light reflected by the eye of the user from the at least one metasurface and converts the received light into an electrical signal. Therefore, beam guidance is enabled by using a characteristic of all solid-state of the metasurface, which has a better accuracy and a higher reliability. In addition, it may form a beam in any direction of the field of view at any time, without being limited by scanning modes or requiring operate in resonance, with the advantages of fast scanning and wide scanning angles.

The embodiments of the present disclosure are described below in conjunction with the accompanying drawings.

1 FIG. 100 100 110 120 130 150 110 120 140 130 illustrates a schematic diagram of an example systemof an eye tracking system according to some embodiments of the present disclosure. The systemincludes a light source, a metasurface, a detector, and a controller. Light emitted by the light sourceis received and reflected by the metasurface. The reflected light is reflected again by an eye of a userand reaches the detector.

110 110 100 110 110 110 The light sourcemay be an infrared or near-infrared light source. Due to the invisibility of infrared light, the use of infrared light sources in the eye tracking system does not interfere with the eye of the user and may accurately distinguish a boundary between a pupil and an iris, as well as the iris and a sclera in the eye. In some embodiments, the light sourcemay use a Vertical Cavity Surface Emitting Laser (VCSEL) chip to enable the emission of the near-infrared laser beam. The VCSEL chip includes an array of emission units, each emission unit emitting a separate laser beam. The laser beams emitted by the plurality of emission units are incoherent with each other and are arranged dispersedly in space. In some embodiments, the systemalso includes a beam adjuster configured to adjust a direction or a shape of the beam emitted by the light source. The beam adjuster, for example a collimating lens, configured to receive the light emitted by the light source, collimate it, and then emit it. The beam adjuster, for example, is a Diffractive Optical Element (DOE), configured to receive the light emitted by the light source, homogenize it, and then emit it.

120 150 120 120 120 120 120 120 110 150 120 140 The metasurfacemay be implemented as a tunable metasurface. The controllermay dynamically regulate optical parameters (for example amplitude, phase, polarization) of the metasurfacebased on schemes such as mechanical driving, modulation of carrier concentration, phase change materials, and the like. For example, structure units in the metasurfaceare driven by microelectromechanical or nanomechanical systems, the physical shape and spatial arrangement of these structural units may be reconfigured to enable dynamic controlling. For example, dynamic controlling may be enabled by changing the carrier concentration of structural units in the metasurface. For example, phase change materials are induced to undergo phase changes by external excitation, and the difference in a refractive rate between different phases is used for controlling. With these schemes, the tunable metasurfaces may introduce sudden changes of the optical parameters to enable dynamic controlling, thereby breaking the dependence of a traditional optical component on a propagation path. In some embodiments, the metasurfaceis a reflective metasurface. The metasurfaceis used to manipulate the laser beam. Specifically, the metasurfacereceives a near-infrared light from the light source, the controllercontrols the reflection direction of the near-infrared light by controlling the metasurface, so that the near-infrared light is reflected to the eye of the user.

150 110 140 120 150 110 The controllercontrols the light emitted by the light sourceto scan the eyeby controlling the metasurface. In some embodiments, the controllercontrols the light emitted by the light sourceto scan the iris area to obtain the direction of the line of sight and estimate the position of the pupil.

100 130 130 140 120 130 150 140 141 142 143 142 141 143 141 142 140 143 144 143 142 150 150 The systemmay include one or more detectors. The detectormay be a Photo-Diode(PD), configured to receive the light reflected by the eye of the userfrom the metasurfaceand convert the received light into an electrical signal. The detectormay also be a Position Sensing Detector(PSD), configured to provide position information to the controller. The eyeincludes a pupil, an iris, and a cornea. The irisis a colored part surrounding the pupil. The corneais a transparent part covering an outer layer of the pupiland the iris. Light reflected in different ways may be generated as the light shines on different parts of the eye. For example, when the light shines on the cornea, a specular reflection occurs and a light spotis formed on the cornea. For example, when the light shines on the iris, a diffuse reflection occurs. According to such reflection characteristics, the controllermay determine the position of the pupil/iris based on an area or range of the light of the diffuse reflection, and the controllermay also determine the position of the cornea based on the position of the light spot of the specular reflection.

100 160 160 100 120 130 160 160 In some embodiments, the systemalso includes a lens. The lensmay be a VR display lens or include an AR lens, thus enabling the systemto be applied in the field of VR or AR. The metasurfaceand the detectormay be arranged along the lensor within the lens.

2 FIG. 200 200 150 120 210 150 110 130 230 130 140 150 220 illustrates a schematic diagram of an example architectureof controlling an eye tracking system according to some embodiments of the present disclosure. In the architecture, the controllercontrols the metasurfaceby using a scanning signal. Furthermore, the controllercontrols the light sourceand the detectorby using a modulation signal. Furthermore, the detectorreceives the light reflected by the eye of the userand converts it into an electrical signal, and the controllerprocesses the electrical signal. The control procedure and the processing procedure will be described below respectively.

120 150 120 120 Depending on the specific implementation of the metasurface, the controllermay use an appropriate control approach. In some embodiments, the metasurfacemay be implemented as an electrical tunable metasurface. By changing the carrier concentration in the metasurface, the phase shift may be controlled, thereby enabling a control of the reflection direction of the light.

3 FIG. 3 FIG. 300 300 300 310 320 illustrates a schematic diagram of an exampleof a metasurface according to some embodiments of the present disclosure. The exampleis an electrical tunable metasurface of Transparent Conductive Oxide (TCO) materials combined with a metal structure.illustrates a cross-sectional view of the electrically tunable metasurface. The exampleincludes a substrateand a structural unit.

310 310 301 302 303 301 320 302 302 303 320 303 2 3 The substratemay include a plurality of layers of materials for providing electrical isolation, accumulated carriers, physical support, and the like. As an example, the substrateincludes an insulation layer, a TCO layerand a substrate layer. The insulation layer, for example, uses aluminum oxide (AlO) to electrically insulate the structural unitfrom the TCO layer. The TCO layerhas a conductivity close to that of a metal, for example using Indium Tin Oxide (ITO) material. It has a near zero dielectric constant point in the near-infrared band. The substrate layer, for example, uses gold (Au). The structural unitmay also be referred to as a nanoantenna, for example, by using the metal materials Au, which is the same as the substrate layer.

3 FIG. bias bias 320 303 330 150 210 320 300 320 150 By applying an external electric field, the concentration of free carriers in conductive materials may be changed, thereby implementing the electrical tunable metasurfaces. In the example shown in, applying a bias voltage Vbetween the structural unitand the substrate layermay form a charge accumulation layer. By adjusting the bias voltage V, the carrier concentration may be increased, thereby enabling the control of the reflection direction of the light. Based on such a principle, the controllerprovides, by using the scanning signal, the bias voltage to the structural unit. The exampleonly includes one structural unit, and for a metasurface that includes the plurality of structural units, the controllermay provide a one-dimensional bias scheme and a two-dimensional bias scheme.

4 FIG. 400 400 410 420 430 401 420 410 430 430 420 420 150 210 430 420 420 400 illustrates a schematic diagram of an exampleof a one-dimensional metasurface according to some embodiments of the present disclosure. The one-dimensional metasurface refers to a metasurface using a one-dimensional bias scheme. The exampleincludes a substrate, a structural unit, and a control line. A substrate layerin both the structural unitand the substrateare made of metals, for example gold, silver, aluminum, etc. The materials of the two may or may not be the same. The substrate layer is grounded. The control lineis a metal wire. Each control lineextends in the Y direction to connect the plurality of structural units, thereby forming a metasurface arranged linearly and periodically in a one dimension. The array period and height of the structural unitare both sub-wavelength scales. The controllertransmits the scanning signalby each control lineto simultaneously provide the bias voltage to the plurality of structural unitsconnected to it, thereby enabling the one-dimensional bias scheme. It would be appreciated that the plurality of structural unitsin the examplemay be arranged regularly with equal spacing or alternately, and no limitation is suggested in this disclosure.

5 FIG. 500 500 510 520 530 501 520 530 510 501 530 520 520 530 150 520 520 500 illustrates a schematic diagram of an exampleof a two-dimensional metasurface according to some embodiments of the present disclosure. The two-dimensional metasurface is a metasurface that adopts a two-dimensional bias scheme. The exampleincludes a substrate, a structural unit, and a control line. A substrate layer, the structural unit, and the control linein the substrateare all made of metals, for example gold, silver, copper, aluminum, and the like. The materials of the three may or may not be the same. The substrate layeris grounded. Each control linemay extend in the Y and X directions and connect to the structural unit, thereby forming a metasurface arranged linearly and periodically in two-dimensions. The array period and height of the structural unitare both sub-wavelength scales. The control lineis a metal wire. The controllermay provide, using scanning signals, the bias voltage to each structural unit, enabling more precise control of the metasurface. It would be appreciated that the plurality of structural unitsin the examplemay be arranged regularly with equal spacing or alternately, and no limitation is suggested in this disclosure.

150 1 FIG. For the eye tracking system, it is necessary to direct the light towards the direction of the eye of the user. Whether using the one-dimensional or two-dimensional metasurface, the controllermay enable a control of the reflected light in the two-dimensional direction by controlling the bias voltage applied at the metasurface, thereby enabling the light scanning of the eye of the user (as shown in the curved dashed arrow in).

6 FIG. 600 610 600 110 150 620 150 630 150 150 640 illustrates a schematic diagram of an exampleof a reflection direction of light from a metasurface according to some embodiments of the present disclosure. A structural unitin the examplereceives and reflects light emitted from the light source. The controllermay control the reflection direction of the light to rotate around the X-axis. For example, the reflection directioncorresponds to a pitch angle of θ. The controllermay also control the reflection direction of the light to rotate around the Z-axis. For example, the reflection directioncorresponds to a yaw angle of φ. Through these two approaches, the controllerenables a control of the reflected light in a one-dimensional direction. The controllermay also control the reflection direction of the light to rotate around both the X and Z axes simultaneously, for example, the reflection direction, thereby enabling a control of the reflected light in the two-dimensional direction.

120 150 210 150 220 130 2 FIG. The control of the metasurfaceby the controllerusing the scanning signalhas been described above in various embodiments. Returning to, alternatively, or in addition, the controllerprocesses the electrical signaloutput by the detectorto determine the pupil position and the direction of the line of sight.

130 As described above, the position of the pupil/iris may be determined based on the diffuse reflection of the light, and the position of the cornea may be determined based on the specular reflection of the light. In order to separate the diffuse reflection of the light and the specular reflection of the light, different detectorsmay be used.

7 FIG. 700 700 200 130 705 715 illustrates a schematic diagram of an example architectureof controlling an eye tracking system according to some embodiments of the present disclosure. The example architectureis extended on the basis of the example architecture. For example, the detectorfurther includes a first detectorand a second detector.

705 140 120 150 1 705 110 150 725 730 710 705 735 765 705 150 740 765 The first detectormay be, for example, a diffuse reflection detector. During a scanning process of the eyeby the metasurfacecontrolled by the controllerfor one period, the first light Lof the diffuse reflection is continuously received by the first detector. Such light may include an infrared light from the light sourceor an ambient light. The controllerperforms peak signal detectionand signal processing(for example, noise removal) on the first electrical signaloutput by the first detectorto generate pixel-level pupil/iris images. Pixels in the image with electrical signals peaks below the first threshold correspond to the pupil area, while pixels with the peaks above the first threshold correspond to the iris area. Based on this image, eye feature detectionmay be performed, and the position of the pupil center and the size of the pupil may be determined according to the detected eye features. Accurate pupil information may be obtained by using a smaller number of (for example, 1 to 2) first detectors. This facilitates system miniaturization and power consumption reduced. In some embodiments, the controllermay determinethe pupil position and the direction of the line of sight by using relevant algorithms, based on the results of eye feature detection(i.e., the detected eye features). The pupil position and the direction of the line of sight may be used for eye tracking, identity recognition, and the like.

735 770 150 750 120 110 140 150 735 150 140 In some embodiments, a spot/pupil/iris imagemay be used for eye image reconstruction. For example, the controllercontrols, by using the scanning signal, the reflection direction of the light emitted by the metasurfaceto the light source, and scans the eyeby the reflected light. The controllerreconstructs a three-dimensional image of the eye based on the generated spot/pupil/iris imageduring the scanning process. Alternatively, or in addition, the controllermay implement the scanning of the eyebased on the predetermined scanning mode. The scanning mode is determined based on parameters such as a scanning route, a scanning range, or a light intensity and/or the like.

2 715 715 150 735 720 715 150 120 140 755 715 715 2 150 150 The second light Lreflected by the mirror is received directionally by the second detector. The second detectoris, for example, a light-emitting diode. The controllermay generate an imageof a light spot on the cornea according to the second electrical signaloutput by the second detector. The pixel in the image with a peak electrical signal greater than the second threshold corresponds to the center position of the spot. As the central position of a corneal curvature is on a bisector (normal) of the angle between the direction of the incident light and the direction of the reflected light, the controlleruses the scanning signal to control the bias voltage of the metasurface, thereby controlling the reflection direction of the light, i.e. the incident direction reaching the eye. Therefore, based on position informationin the scanning signal, the direction of the incident light on the cornea may be determined. The second detectormay only receive the reflected light from the cornea directionally, and therefore the direction of the reflected light from the cornea may be determined based on the physical position of the second detector. In some embodiments, the second reflected light Lis received by a position sensing detector. The position sensing detector is an optical detector that may measure the continuous position of the light points on the detector surface, and may convert the position of the light points on the photosensitive surface into electrical signals. Such a detector may provide position information to the controllerto replace the physical position of the detector. Therefore, during system calibration, there is no need to calibrate and store the physical position of the detector. During eye tracking, the controllerdetermines the direction of the reflected light of the corneal by the position information in the electrical signal output by the position sensing detector, in order to finally determine the direction of the line of sight.

715 In some embodiments, if the radius of the cornea is known, the position of the center of the cornea may be determined according to the distance from the center of the cornea to the spot of the light being equal to the radius of the cornea. That is, the position of the center of the cornea may be determined from a single spot detected by a single second detector. This greatly reduces the number of detectors required and facilitates miniaturization and lower power consumption of the system.

2 715 150 In other embodiments, if the radius of the cornea is an unknown quantity, the plurality of second light Lreflected from the specular surface is sequentially received by the plurality of second detectorsarranged dispersedly. The controllerdetermines a plurality of normal directions according to the center position of these light spots, the direction of the incident light, and the direction of the reflected light. Furthermore, it determines the position of the center of the cornea according to an intersection point of the plurality of normal directions. Considering a curvature of the cornea, a very accurate corneal center position may be obtained by using a relatively small number of (e.g., 3 to 4) light-emitting diodes. This facilitates miniaturization and reduced power consumption of the system.

150 140 In this way, according to the line connecting the center position of the pupil and the center position of the cornea, the controllermay determine the direction of line of sight of the eye of the user, thereby enabling the eye tracking function.

740 150 745 150 750 140 745 In some embodiments, based on the output of determiningthe pupil position and the direction of the line of sight, the controllermay generate an area of interest (ROI). The controllermay adjust the scanning range of the scanning signalfor the eye of the userbased on the area of interest, thereby improving the speed and efficiency of eye tracking.

2 FIG. 7 FIG. 150 110 130 230 150 110 705 715 760 110 For the eye tracking system, continuously pursuing small size and low power consumption facilitates device miniaturization. Alternatively, or in addition, in the example of, the controllercontrols the light sourceand the detectorby using the modulation signal. In the example of, the controllercontrols the light source, the first detector, and the second detectorby using the modulation signal. While controlling the light sourceto emit light, turning on the corresponding detector for reception facilitates reduction of system power consumption.

2 FIG. 2 FIG. 150 230 110 110 150 130 130 140 120 110 130 230 150 110 230 130 In the example of, the controlleruses the modulation signalto provide a driving voltage to the light sourceto control the light sourceto emit light. Synchronously, the controlleruses the modulation signal to provide a bias voltage to the detectorto enable the detectorto receive light reflected by the eye of the userfrom the metasurface. If the light sourcedoes not emit light, the detectoris turned off, thereby reducing system power consumption. It would be appreciated that although not shown in, the modulation signalprovided by the controllermay be converted by any suitable driving circuit before providing the driving voltage to the light source, and the modulation signalmay also be converted by any suitable switching circuit before providing the bias voltage to the detector.

7 FIG. 7 FIG. 150 110 760 110 150 705 760 705 1 1 110 150 705 705 150 715 760 715 2 715 2 120 715 720 760 150 110 760 705 715 In the example of, the controllerprovides the driving voltage to the light sourceby using the modulation signalto control the light sourceto emit light. Synchronously, the controllerprovides the bias voltage to the first detectorby using the modulation signal, to turn on the first detectorto receive the first light L. As described above, the first light Lmay include an infrared light from the light sourceor may include an ambient light. Alternatively, or in addition, the controllercontinuously provides the bias voltage to the first detectorto ensure that the first detectoris always on. Further, the controllerprovides the bias voltage to the second detectorby using the modulation signal, to turn on the second detectorto receive the second light L. As described above, the second detectorreceives directionally the second light Lreflected from the specular surface of the cornea. Thus, it is possible to turn on the second detector again when the reflected light from the metasurfacearrives at a corresponding position of or near the cornea, and to determine the corresponding position as the center position of the light spot when the peak value of the second electrical signaloutput therefrom is greater than the second threshold. It would be appreciated that although not shown in, the modulation signalprovided by the controllermay be converted by any suitable driving circuitry before providing the driving voltage to the light source, and the modulation signalmay also be converted by any suitable switching circuitry before providing the bias voltage to the first detectorand the second detector.

150 150 The control approach for the eye tracking system have been described in the various embodiments described above. It would be appreciated that although the above control approaches are described by the controlleras the executor, these control approaches may be implemented by different controllers, or the eye tracking system may include the plurality of controllersfor controlling the light source, the metasurfaces, the detectors, and the various signal processing, and the like, respectively.

The embodiments of the present disclosure also provide a wearable display device, which may include the eye tracking system described above.

8 FIG. 800 800 800 illustrates a schematic diagram of an exampleof a wearable display device according to some embodiments of the present disclosure. The wearable display device, for example may be an AR, VR or MR display device, including, but not limited to, a head-mounted display, a smart glasses, a user behavior analysis device, and/or the like. The position relationship of various components in the wearable display devicewill be described below from a structural perspective.

800 110 120 130 160 160 130 160 140 140 130 160 110 120 160 160 160 800 110 120 As an example of components corresponding to a single eye, in some embodiments, the wearable display deviceincludes the light source, the metasurface, the plurality of detectors, and the lens. The lensis, for example, a VR display lens or includes an AR lens. The plurality of detectorsare located near an edge of the lensso as not to obstruct the line of sight, but also to be located as close as possible to the eye of the user, for example a right eye, to facilitate reception of the light reflected from the eye. Further, the plurality of detectorsmay be arranged around the lens. The light sourceand the metasurfaceare located near an end of the lensalong the horizontal axis H of the lens. In the case that the lensis a common elliptical shape, the horizontal axis H may typically be a long axis of an ellipse. In the case that the user wears the wearable display device, the light sourceand the metasurfaceare, for example, close to a nose of the user.

800 810 820 110 120 160 160 810 820 160 160 800 110 120 810 820 Continuing with the example of the components corresponding to a single eye, in some embodiments, the wearable display devicefurther includes a light sourceand a metasurface. The light sourceand the metasurfaceare located near one of the end of the lensalong the horizontal axis H of the lens, and the light sourceand the metasurfaceare located near the other end of the lensalong the horizontal axis H of the lens. In the case that a user wears the wearable display device, the light sourceand the metasurfaceare, for example, close to a nose of the user, the light sourceand the metasurfaceare, for example, close to a temple of the user.

9 FIG. 1 FIG. 500 900 100 900 illustrates a flowchart of a processfor eye tracking according to some embodiments of the present disclosure. The processmay be implemented at the system. The processis described below with reference to.

100 At block, the systemcontrols at least one light source to emit light.

At block, control at least one metasurface to reflect the light emitted by the at least one light source to the eye of the user.

100 At block, the systemcontrols at least one detector to receive the light reflected by the eye of the user from the at least one metasurface.

940 100 At block, the systemconverts the received light into an electrical signal.

100 In some embodiments, the systemcontrols, by using a scanning signal, a bias voltage of a structural unit in the at least one metasurface to control the reflection direction of the light.

100 In some embodiments, the systemdetermines a pupil position and/or line of sight direction of the eye of the user by at least one of: position information in the scanning signal, the electrical signal, and a position of the detector that outputs the electrical signal.

100 In some embodiments, the at least one detector includes a first detector, the systemcontrols the first detector to receive a first light reflected from the at least one metasurface by a pupil and/or an iris of the eye of the user; converts the first light into a first electrical signal; and determines, at least based on the first electrical signal, the pupil position of the eye of the user.

100 In some embodiments, at least one detector includes a plurality of second detectors, the systemcontrols the plurality of second detectors to receive a second light from the at least one metasurface reflected by a cornea of the eye of the user; converts the second light into a second electrical signal; and determines, at least based on the second electrical signal, the direction of line of sight of the eye of the user.

100 In some embodiments, the systemcontrols the plurality of second detectors to provide position information of the detector.

10 FIG. 1 FIG. 1000 1000 100 1000 illustrates a flowchart of a processfor eye tracking according to some embodiments of the present disclosure. The processmay be implemented at the system. The processis described below with reference to.

1010 100 At block, the systemcontrols, by using a modulation signal, at least one light source to emit light.

1020 100 At block, the systemcontrols at least one metasurface to reflect the light emitted by the at least one light source to an eye of the user.

1030 100 At block, the systemcontrols, by using the modulation signal, at least one detector to receive light from the at least one metasurface reflected by the eye of the user.

1040 100 At block, the systemconverts the received light into an electrical signal.

100 In some embodiments, the at least one light source comprises a first light source and a second light source, and the at least one metasurface comprises a first metasurface associated with the first light source and a second metasurface associated with the second light source. The systemcontrols, by using different modulation signals with a phase difference, the first light source and the second light source emit light respectively; and controls, by using the different modulation signals with the phase difference, the at least one detector to receive light from the first metasurface and the second metasurface reflected by the eye of the user simultaneously.

100 In some embodiments, at least one detector includes a first detector, the systemcontrols the first detector to receive the first light reflected from the at least one metasurface by the pupil and/or the iris of the eye of the user; converts the first light into the first electrical signal; and determines, at least based on the first electrical signal, the pupil position of the eye of the user.

100 In some embodiments, at least one detector includes a plurality of second detectors, the systemcontrols the plurality of second detectors to receive the second light from the at least one metasurface reflected by the cornea of the eye of the user; converts the second light into a second electrical signal; and determines, at least based on the second electrical signal, the direction of line of sight of the eye of the user.

100 In some embodiments, the systemcontrols the plurality of second detectors to provide the position information of the detector.

11 FIG. 11 FIG. 11 FIG. 1 FIG. 1100 1100 1100 100 illustrates a block diagram of an example electronic devicein which one or more embodiments of the present disclosure may be implemented. It would be appreciated that the electronic deviceshown inis only an example and should not constitute any restriction on the function and scope of the embodiments described herein. The electronic deviceshown inmay be used to implement the systemof.

11 FIG. 1100 1100 1110 1120 1130 1140 1150 1160 1110 1120 1100 As shown in, the electronic deviceis in the form of a general computing device. The components of the electronic devicemay comprise, but are not limited to, one or more processors or processing units, a memory, a storage device, one or more communication units, one or more input devices, and one or more output devices. The processing unitsmay be actual or virtual processors and may execute various processes according to the programs stored in the memory. In a multiprocessor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device.

1100 1100 1120 1130 1100 The electronic devicetypically comprises a variety of computer storage media. Such media may be any available media that is accessible to the electronic device, comprising but not limited to volatile and non-volatile media, removable and non-removable media. The memorymay be volatile memory (such as registers, caches, random access memory (RAM)), nonvolatile memory (such as a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory), or some combination thereof. The storage devicemay be any removable or non-removable medium, and may comprise machine-readable medium, such as a flash drive, a disk, or any other medium which may be used to store information and/or data (for example training data for training) and may be accessed within the electronic device.

1100 1120 1125 11 FIG. The electronic devicemay further comprise additional removable/non-removable, volatile/non-volatile storage medium. Although not shown in, a disk driver for reading from or writing to a removable, non-volatile disk (such as a “floppy disk”), and an optical disk driver for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each driver may be connected to the bus (not shown) by one or more data medium interfaces. The memorymay comprise a computer program product, which comprises one or more program modules configured to execute various methods or actions of the various embodiments disclosed herein.

1140 1100 1100 The communication unitimplements communication with other computing devices via a communication medium. In addition, functions of components in the electronic devicemay be implemented by a single computing cluster or multiple computing machines, which may communicate through a communication connection. Therefore, the electronic devicemay be operated in a networking environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

1150 1160 1100 1140 1100 1100 The input devicemay be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output devicemay be one or more output devices, such as a display, a speaker, a printer, etc. The electronic devicemay also communicate with one or more external devices (not shown) through the communication unitas required. The external device, such as a storage device, a display device, etc., communicate with one or more devices that enable users to interact with the electronic device, or communicate with any device (for example, a network card, a modem, etc.) that makes the electronic devicecommunicate with one or more other computing devices. Such communication may be executed via an input/output (I/O) interface (not shown).

According to example implementation of the present disclosure, there is provided a computer-readable storage medium on which a computer-executable instruction or computer program is stored, wherein the computer-executable instructions are executed by a processor to implement the methods described above.

Various aspects of the present disclosure are described herein with reference to the flow chart and/or the block diagram of the method, the device, the apparatus, and the computer program product implemented in accordance with the present disclosure. It would be appreciated that each block of the flowchart and/or the block diagram and the combination of each block in the flowchart and/or the block diagram may be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to the processing units of general-purpose computers, special computers, or other programmable data processing devices to produce a machine that generates a device to implement the functions/acts specified in one or more blocks in the flow chart and/or the block diagram when these instructions are executed through the processing units of the computer or other programmable data processing devices. These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions enable a computer, a programmable data processing device and/or other devices to work in a specific way. Therefore, the computer-readable medium containing the instructions comprises a product, which comprises instructions operable to implement various aspects of the functions/acts specified in one or more blocks in the flowchart and/or the block diagram.

The computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other devices, so that a series of operational steps may be performed on a computer, other programmable data processing apparatus, or other devices, to generate a computer-implemented process, such that the instructions which execute on a computer, other programmable data processing apparatus, or other devices are operable to implement the functions/acts specified in one or more blocks in the flowchart and/or the block diagram.

The flowchart and the block diagram in the drawings show the possible architecture, functions and operations of the system, the method and the computer program product implemented in accordance with the present disclosure. In this regard, each block in the flowchart or the block diagram may represent a part of a module, a program segment, or instructions, which includes one or more executable instructions for implementing the specified logic function. In some alternative implementations, the functions marked in the block may also occur in a different order from those marked in the drawings. For example, two consecutive blocks may actually be executed in parallel, and sometimes may also be executed in a reverse order, depending on the function involved. It should also be noted that each block in the block diagram and/or the flowchart, and combinations of blocks in the block diagram and/or the flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by the combination of dedicated hardware and computer instructions.

Each implementation of the present disclosure has been described above. The above description provides a number of examples, not exhaustive, and is not limited to the disclosed implementations. Without departing from the scope and spirit of the described implementations, many modifications and changes are obvious to ordinary skill in the art. The selection of terms used in this article aims to best explain the principles, practical application, or improvement of technology in the market of each implementation, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.