Dual detail encoding in a distributed system

Wireless streaming technology is used both within and outside of the video game industry. In the video game industry, some virtual reality (VR) gaming systems utilize wireless streaming technology in order to leverage the high-computing capacity of a host computer for executing a video game, while providing a wearer of a wireless VR headset with greater mobility, as compared to a tethered (non-wireless) headset. Despite these advantages, data rates of existing wireless protocols limit the amount of data that can be sent in a relatively short amount of time. This presents challenges in VR gaming systems that operate at high frame rates, especially when playing graphics intensive video games.

Provided herein are technical solutions to improve and enhance these and other systems.

Described herein are, among other things, techniques, devices, and systems for streaming pixel data from a host computer to a display device using a dual detail encoding scheme. This dual detail encoding scheme allows for transporting pixel data, among other data, from a host computer to a display device in a timely manner and without compromising the perceived fidelity of the imagery that is displayed on the display device. This is particularly beneficial when streaming real-time content, such as video game content. For instance, unlike prerecorded content, real-time content cannot be streamed in advance and buffered on the client until the content is ready to be displayed. Instead, frames are rendered milliseconds before the content is displayed based on data received from the display device.

While the forthcoming description is not limited to gaming systems, or even to VR systems, many of the examples described herein pertain to a VR gaming system that includes a host computer and a HMD communicatively coupled thereto. It is to be appreciated, however, that a HMD is just one example of a “display device” that can be utilized for displaying streamed content, and other types of display devices are contemplated herein. Furthermore, the techniques described herein may be implemented in wireless distributed systems and/or wired distributed systems (e.g., a host computer and a display device connected via a cable). In distributed systems that include a HMD, the HMD may be worn by a user for purposes of immersing the user in a VR environment or an augmented reality (AR) environment, as the case may be. One or more display panels of the HMD are configured to present images based on data generated by an application (e.g., a video game). The application executes on the host computer and generates pixel data for individual frames of a series of frames. The pixel data is encoded and sent (or transmitted) from the host computer to the HMD to present images that are viewed by a user through the optics included in the HMD, making the user perceive the images as if the user was immersed in a VR or AR environment.

In some embodiments, the HMD is configured to send data to the host computer, and the host computer is configured to use the data in various ways, such as to generate the pixel data for a given frame. For example, the host computer may use head tracking data received from the HMD to generate pose data indicative of a predicted pose that the HMD will be in at a time at which light emitting elements of the display panel(s) of the HMD will illuminate for the frame. This pose data may be input to the executing application (e.g., a video game) to generate the pixel data for a given frame. In some examples, as will be described in more detail below, eye tracking data received from the HMD may be used to determine a subregion of a scene that is ultimately displayed with higher quality than a remainder of the displayed image so that the user is looking at a more detailed portion of the image, which makes the image look sharp. In general, the HMD may receive the encoded pixel data from the host computer and display an image on the HMD's display panel(s) based on the received pixel data. As will be described in more detail below, the HMD may reconstruct and/or modify the pixel data, as needed, before rendering the final images so that the user perceives the content as it was intended to be perceived consistent with the user's head orientation and gaze.

As noted above, limited data rates of existing communications protocols (e.g., wireless protocols) present challenges in distributed display systems, especially in distributed, wireless VR gaming system while playing a graphics intensive video game. For instance, depending on the wireless protocol, a typical data rate for transporting data wirelessly from a host computer to a display device may range from 50 to 300 megabits per second (Mbps). Consider an example where the available data rate is on the order of 100 Mbps. In this example, a video game executing on the host computer may be rendering frames at a resolution of 2.5 k pixels (i.e., an image with 2500×2500 pixels) per eye, which equates to 12,500,000 pixels across two images, one image per eye. At 3 bytes per pixel, this means that the amount of data to transport the images for a given frame is on the order of 300 megabits. In a 120 hertz (Hz) VR gaming system, in order to transmit all of the image data in a timely manner, the host computer would therefore have to transmit pixel data at a rate of 36,000 Mbps (or 36 gigabits per second (Gbps)). However, this amount of pixel data cannot be transported fast enough in a distributed system that is constrained by a 100 Mbps data rate.

Described herein are techniques, devices, and systems for implementing a dual detail encoding scheme that allows for transporting pixel data, among other data, from a host computer to a display device in a timely manner and without compromising the perceived fidelity of the imagery that is displayed on the display device. An example process for streaming pixel data from a host computer to a display device may include rendering, by a graphics-rendering application executing on the host computer, a frame for a scene at a first resolution, generating a 2D array of pixels for the frame, encoding the 2D array of pixels to obtain encoded pixel data for the frame, and transmitting the encoded pixel data to the display device. Particularly, the 2D array of pixels that is encoded in the aforementioned process provides two levels of detail per eye. That is, the 2D array of pixels comprises, for each eye, a copy of the frame downscaled to a second resolution less than the first resolution, and a copy of a subregion of the scene. And because the frame includes the subregion, the 2D array of pixels comprises, for each eye, two copies of the subregion (one being a copy of the subregion itself, and another being included in the copy of the downscaled frame). In some examples, the pixels representing the copies of the subregion of the scene correspond one-to-one with original pixels of the frame, which allows for presenting images via the display device where the subregions are rendered with the same or similar quality as they were rendered by the application in the original frame. Furthermore, by transmitting copies of the subregion (the subregion having fewer pixels than the full image) and copies of the downscaled frame, the host computer is able to transmit the reduced amount of pixel data at a data rate that is commonly available in distributed display systems, including wireless systems. Hence, the dual detail encoding techniques described herein allow the host computer to send a much smaller amount of data (i.e., the 2D array of pixels), and the display device is configured to reconstruct the images from the 2D array of pixels such that the user of the display device perceives the presented imagery as high quality imagery (e.g., images that look sharp, crisp, etc.).

An example process for displaying an image based on pixel data received from a host computer may include receiving, from a host computer, encoded pixel data for a frame for a scene, decoding the encoded pixel data to obtain a 2D array of pixels for the frame, enlarging a first copy of the frame based at least in part on first pixels of the 2D array of pixels to obtain a first enlarged copy of the frame, and enlarging a second copy of the frame based at least in part on second pixels of the 2D array of pixels to obtain a second enlarged copy of the frame. The process may continue by generating a first image based at least in part on the first enlarged copy of the frame and third pixels of the 2D array of pixels that represent a first copy of a subregion of the scene, wherein a subset of the third pixels at a periphery of the subregion are blended in the first image, and generating a second image based at least in part on the second enlarged copy of the frame and fourth pixels of the 2D array of pixels that represent a second copy of the subregion, wherein a subset of the fourth pixels at the periphery of the subregion are blended in the second image. The first image and the second image can then be presented on respective display panels of the display device, such as left and right display panels, or on a single display panel.

By reducing the amount of data that is to be transmitted from a host computer to a display device using the dual detail encoding techniques described herein, a distributed display system can stream pixel data in a timely manner for display of corresponding imagery on the display device, and the user perceives the imagery as high-fidelity imagery on the display device. For example, even though the portion of the displayed images outside of the subregion may be presented with relatively low quality (e.g., low resolution), the user's gaze is likely to be directed at the subregion of the scene where the image quality is relatively higher, making it seem, to the user, as though the images are of high quality. In other words, distributed display systems can implement the dual detail encoding techniques described herein in order to avoid transmitting pixel data for areas of the scene that the user is unlikely to look at when the corresponding images are presented, yet the user does not perceive the image to be of lower quality than the originally-rendered frame.

Also disclosed herein are devices, systems, and non-transitory computer-readable media storing computer-executable instructions to implement the techniques and processes disclosed herein. Although the techniques and systems disclosed herein are discussed, by way of example, in the context of video game applications, and specifically VR gaming applications, it is to be appreciated that the techniques and systems described herein may provide benefits with other applications, including, without limitation, non-VR applications (e.g., AR applications, mixed reality (MR) applications), and/or non-gaming applications, such as industrial machine applications, defense applications, robotics applications, and the like.

1 FIG. 1 FIG. 1 FIG. 100 102 104 104 106 102 102 1 102 106 102 102 102 104 106 104 102 104 102 104 102 102 104 102 102 104 102 104 is a diagram illustrating an example distributed display systemincluding a host computer(s)and a display device(s) in the form of a HMD, in accordance with embodiments disclosed herein.depicts the HMDbeing worn by a user.also depicts example implementations of a host computerin the form of a laptop() carried in a backpack, for example, or a personal computer (PC)(N), which may be situated in the user'shousehold, for example. It is to be appreciated, however, that these exemplary types of host computersare non-limiting to the present disclosure. For example, the host computercan be implemented as any type and/or any number of computing devices, including, without limitation, a PC, a laptop computer, a desktop computer, a portable digital assistant (PDA), a mobile phone, tablet computer, a set-top box, a game console, a portable game device, a server computer, a wearable computer (e.g., a smart watch, etc.), or any other electronic device that can transmit data to, and receive data from, other devices. The host computermay be collocated in the same environment as the HMD, such as a household of the userwearing the HMD. Alternatively, the host computermay be remotely located with respect to the HMD, such as a host computerin the form of a server computer that is located in a remote geographical location with respect to the geographical location of the HMD. In a remote host computerimplementation, the host computermay be communicatively coupled to the HMDvia a wide-area network, such as the Internet. In a local host computerimplementation, the host computermay be collocated in an environment (e.g., a household) with the HMD, whereby the host computerand the HMDmay be communicatively coupled together either directly or over a local area network (LAN) via intermediary network devices.

104 104 104 102 It is also to be appreciated that the exemplary HMDis but one example type of display device that can be implemented with the techniques and systems described herein. For example, other types and/or numbers of display devices can be utilized in lieu of, or in conjunction with, the HMDdescribed in many of the examples herein. Accordingly, the HMDmay be referred to herein more generally as a “display device,” and it is to be appreciated that other suitable types of display devices may exchange data with the host computer. Such display devices may include, without limitation, a television, a portable game device with a display, a laptop computer, a desktop computer, a PDA, a mobile phone, a tablet computer, a wearable computer (e.g., a smart watch, etc.), or any other electronic device that can transmit/receive data as well as display imagery on a display screen(s) or panel(s).

1 FIG. 104 102 108 104 104 104 110 112 112 110 114 114 110 110 In the example of, the HMDand the host computerare communicatively coupled together and are configured to work together in a collaborative fashion to render a given frame and present a corresponding image(s) on a display panel(s)of the HMD. This collaboration iterates over a series of frames to render a series of images (e.g., images for a VR game) on the HMD. In the illustrated implementation, the HMDincludes one or more processorsand memory(e.g., computer-readable media). In some implementations, the processors(s)may include a central processing unit (CPU)(s), a graphics processing unit (GPU)(s), both CPU(s) and GPU(s), a microprocessor, a digital signal processor or other processing units or components known in the art. Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), etc. Additionally, each of the processor(s)may possess its own local memory, which also may store program modules, program data, and/or one or more operating systems. In some embodiments, each of the processor(s)may possess its own encoder and/or decoder hardware.

112 112 110 112 110 The memorymay include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk ROM (CD-ROM) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, redundant array of independent disks (RAID) storage systems, or any other medium which can be used to store the desired information and which can be accessed by a computing device. The memorymay be implemented as computer-readable storage media (CRSM), which may be any available physical media accessible by the processor(s)to execute instructions stored on the memory. In one basic implementation, CRSM may include RAM and Flash memory. In other implementations, CRSM may include, but is not limited to, ROM, EEPROM, or any other tangible medium which can be used to store the desired information and which can be accessed by the processor(s).

104 112 110 116 112 110 112 110 116 108 104 116 102 116 116 104 104 104 108 104 In general, the HMDmay include logic (e.g., software, hardware, and/or firmware, etc.) that is configured to implement the techniques, functionality, and/or operations described herein. The computer-readable mediacan include various modules, such as instruction, datastores, and so forth, which may be configured to execute on the processor(s)for carrying out the techniques, functionality, and/or operations described herein. An example functional module in the form of a compositoris shown as being stored in the computer-readable mediaand executable on the processor(s), although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SOC), and/or other logic. Furthermore, additional or different functional modules may be stored in the computer-readable mediaand executable on the processor(s). The compositoris configured to modify pixels (or pixel data) to generate an image(s) and to output the modified pixel data representing the image(s) to a frame buffer (e.g., a stereo frame buffer) so that the image(s) can be presented on the display panel(s)of the HMD. For example, the compositoris configured to reconstruct the frame using the pixel data received from the host computer, as will be described in more detail below. Furthermore, the compositormay apply other modifications to the pixel data before outputting the final pixel data to the frame buffer. For example, the compositormay be configured to make adjustments for geometric distortion, chromatic aberration, re-projection, and the like. At least some of these adjustments may compensate for the distortion of the near-to-eye optical subsystem (e.g., lenses and other optics) of the HMD, with other adjustments, such as re-projection adjustments, may compensate for slight inaccuracies in an original pose prediction of the HMDand/or to compensate for the application failing to make frame rate and/or to compensate for a late-arriving data packet(s). For example, a re-projected frame can be generated using pixel data from an application-rendered frame by transforming (e.g., through rotation and re-projection calculations) the application-rendered frame in a way that accounts for an updated prediction of the pose of the HMD, which may be more accurate than the original pose prediction. Accordingly, the modified pixel data obtained from applying the re-projection adjustments (and/or other adjustments) may be used to present an image(s) on the display panel(s)of the HMDfor the given frame, and this process may iterate over the series of frames.

104 118 118 104 104 106 118 118 The HMDmay further include a head tracking systemthat generates head tracking data. The head tracking systemmay leverage one or more sensors (e.g., infrared (IR) light sensors mounted on the HMD) and one or more tracking beacon(s) (e.g., IR light emitters collocated in the environment with the HMD) to track head motion or movement, including head rotation, of the user. This example head tracking systemis non-limiting, and other types of head tracking systems(e.g., camera-based, inertial measurement unit (IMU)-based, etc.) can be utilized.

104 120 120 104 120 104 104 120 106 120 The HMDmay further include an eye tracking systemthat generates eye tracking data. The eye tracking systemmay include, without limitation, a camera or other optical sensor inside the HMDto capture image data (or information) of a user's eyes, and the eye tracking systemmay use the captured data/information to determine motion vectors, interpupillary distance, interocular distance, a 3D position of each eye relative to HMD, including a magnitude of torsion and rotation (i.e., roll, pitch, and yaw) and gaze directions for each eye. In one example, infrared light is emitted within HMDand reflected from each eye. The reflected light is received or detected by a camera of the eye tracking systemand analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of a usercan be used by the eye tracking system.

1 FIG. 104 122 102 122 122 124 104 102 104 124 104 102 124 124 102 124 104 104 104 124 104 124 104 124 102 depicts the HMDsending datato the host computer. This datacan include, without limitation, the head tracking data and/or the eye tracking data, as described above. Furthermore, the datacan be transmitted via a communications interface(s)of the HMDduring runtime, as frames are being rendered at the host computerand as corresponding imagery is being presented at the HMD. The communications interface(s)of the HMDmay include wired and/or wireless components (e.g., chips, ports, etc.) to facilitate wired and/or wireless data transmission/reception to/from the host computer, either directly or via one or more intermediate devices, such as a wireless access point (WAP). For example, the communications interface(s)may include a wireless network interface controller. The communications interface(s)may conform to the Institute of Electrical and Electronics Engineers (IEEE) 502.11 standard(s) and may include a radio (e.g., having one or more antennas) to facilitate a wireless connection with the host computerand/or other devices and for transmitting and receiving data packets using radio frequency (RF) communication. The communications interface(s)may be built into the HMDor coupled to the HMD(e.g., wireless adapter, peripheral, accessory, etc.) to adapt the HMDto operate as an 502.11-compliant wireless communication device. In some embodiments communications interface(s)may comprise a Universal Serial Bus (USB) wireless adapter (e.g., a USB dongle) that is plugged into a USB port on the HMD. In other embodiments, the communications interface(s)may be coupled to electrical components of the HMDvia a Peripheral Component Interconnect (PCI) interface. It is to be appreciated that the communications interface(s)may further include one or more physical ports to facilitate a wired connection (e.g., via a cable) with the host computerand/or another device(s) (e.g., a plug-in network device that communicates with other wireless networks).

102 102 126 128 128 126 130 130 126 126 114 104 130 102 114 104 100 102 Turning to the host computer, the host computeris shown as including one or more processorsand memory(e.g., computer-readable media). In some implementations, the processors(s)may include a CPU(s), a GPU(s), both CPU(s) and GPU(s), a microprocessor, a digital signal processor or other processing units or components known in the art. Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include FPGAs, ASICs, ASSPs, SOCs, etc. Additionally, each of the processor(s)may possess its own local memory, which also may store program modules, program data, and/or one or more operating systems. In some embodiments, each of the processor(s)may possess its own encoder and/or decoder hardware. Compared to the GPU(s)of the HMD, the GPU(s)of the host computermay be a higher-performance GPU(s) with higher computing capacity. For example, the computing capacity of the GPU(s)of the HMDmay be insufficient for processing graphics for certain graphics intensive video games, which is why the systemis distributed; to leverage the relatively higher computing capacity of the host computerfor executing a graphics intensive video game.

128 102 128 126 128 126 The memoryof the host computermay include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium which can be used to store the desired information and which can be accessed by a computing device. The memorymay be implemented as CRSM, which may be any available physical media accessible by the processor(s)to execute instructions stored on the memory. In one basic implementation, CRSM may include RAM and Flash memory. In other implementations, CRSM may include, but is not limited to, ROM, EEPROM, or any other tangible medium which can be used to store the desired information and which can be accessed by the processor(s).

102 128 126 132 132 132 1 102 132 102 132 1 1 FIG. In general, the host computermay include logic (e.g., software, hardware, and/or firmware, etc.) that is configured to implement the techniques, functionality, and/or operations described herein. The computer-readable mediacan include various modules, such as instructions, datastores, and so forth, which may be configured to execute on the processor(s)for carrying out the techniques, functionality, and/or operations described herein. An example functional module in the form of an application(s)is shown in. The application(s)may represent a graphics-rendering application, or a graphics-based application, such as a video game(s)(). In some examples, the host computermay include a client application or other software (e.g., a video game client) that is configured to execute the one or more applications. For example, the host computermay have installed thereon gaming software to play video games().

134 102 104 134 134 104 134 102 102 134 102 134 102 134 104 In general, a communications interface(s)of the host computermay include wired and/or wireless components (e.g., chips, ports, etc.) to facilitate wired and/or wireless data transmission/reception to/from the HMD(or any other type of display device), either directly or via one or more intermediate devices, such as a WAP. In some examples, the communications interface(s)may include a wireless network interface controller. The communications interface(s)may conform to the IEEE 502.11 standard(s) and may include a radio (e.g., having one or more antennas) to facilitate a wireless connection with the HMDand/or other devices and for transmitting and receiving data packets using RF communication. The communications interface(s)may be built into or coupled to the host computer(e.g., a wireless adapter, peripheral, accessory, etc.) to adapt the host computerto operate as an 502.11-compliant wireless communication device. In some embodiments, the communications interface(s)may comprise a USB wireless adapter (e.g., a USB dongle) that is plugged into a USB port on the host computer. In other embodiments, the communications interface(s)may be coupled to electrical components of the host computervia a PCI interface. It is to be appreciated that the communications interface(s)may further include one or more physical ports to facilitate a wired connection (e.g., via a cable) with the HMDand/or another device(s) (e.g., a plug-in network device that communicates with other wireless networks).

104 132 1 132 1 104 106 106 108 104 106 104 104 In some embodiments, the HMDmay represent a VR headset for use in VR systems, such as for use with a VR gaming system, in which case the video game() may represent a VR video game(). However, the HMDmay additionally, or alternatively, be implemented as an AR headset for use in AR applications, a MR headset for use in MR applications, or a headset that is usable for VR, AR, and/or MR applications that are not game-related (e.g., industrial applications). In AR, a usersees virtual objects overlaid on a real-world environment, whereas, in VR, the userdoes not typically see a real-world environment, but is fully immersed in a virtual environment, as perceived via the display panel(s)and the optics (e.g., lenses) of the HMD. It is to be appreciated that, in some VR systems, pass-through imagery of the real-world environment of the usermay be displayed in conjunction with virtual imagery to create an augmented VR environment in a VR system, whereby the VR environment is augmented with real-world imagery (e.g., overlaid on a virtual world). Examples described herein pertain primarily to a VR-based HMD, but it is to be appreciated that the HMDis not limited to implementation in VR applications.

132 102 132 132 1 132 108 104 100 104 102 108 104 102 104 132 In general, and as mentioned above, the application(s)executing on the host computercan represent graphics-based application(s)(e.g., a video game()). An applicationis configured to generate pixel data for a series of frames, and the pixel data is ultimately used to present corresponding images on the display panel(s)of the HMD. In a systemwith a HMD, a component of the host computermay determine a predicted “illumination time” for the frame. This predicted “illumination time” for the frame represents a time at which light emitting elements of the display panel(s)of the HMDwill illuminate for the frame. This prediction can account for, among other things, the estimated time it will take to transmit data over a wireless communication link between the host computerand the HMD, as well as a predicted render time of the applicationand/or a known scan-out time of the pixels from the frame buffer(s). In some embodiments, the prediction may be different for a wireless communication link than it is for a wired communication link. In some embodiments, an illumination time can be predicted as an amount of time in the future (e.g., about 20-40 milliseconds in the future), and the amount of time may vary depending on the connection type (e.g., wired connection verses wireless connection).

102 104 122 122 104 102 104 122 132 132 108 104 132 108 104 104 108 132 108 108 108 The host computermay receive, from the HMD, the data(e.g., head tracking data, eye tracking data, etc.). The datamay be generated and/or sent at any suitable frequency, such as a frequency corresponding to the target frame rate and/or the refresh rate of the HMD, or a different (e.g., faster) frequency, such as 360 hertz (Hz) (or 1 sensor reading every 2.7 milliseconds). A component of the host computermay determine a predicted pose that the HMDwill be in at the predicted illumination time based at least in part on head tracking data included in the data. Pose data indicative of the predicted pose may be provided to the executing applicationfor rendering the frame (e.g., generating pixel data for the frame) based on the predicted pose, and the applicationmay output pixel data associated with the frame. This pixel data may correspond to an array of pixels of the display panel(s)of the HMD. For example, the pixel data that is output by the applicationbased on the pose data may include a two-dimensional array of per-pixel values (e.g., color values) for the array of pixels on the display panel(s)of the HMD. In an illustrative example, the HMDmay include a stereo pair of display panels, and the applicationmay render a frame for a scene at a first resolution. For instance, the first resolution at which the frame for each display panelis rendered may be an array of 2500×2500 pixels. In this illustrative example, the pixel data for an individual frame for a display panelmay include 2500×2500 pixel values (or 12,500,000 pixel values for both display panels). In some embodiments, the pixel data may include data for each pixel that is represented by a single set of color and alpha values (e.g., one color value for a red channel, one color value for a green channel, one color value for a blue channel, and one or more values for one or more alpha channels).

102 108 104 104 104 136 136 104 108 104 104 132 104 104 104 104 102 104 It is to be appreciated that logic of the host computermay also generate extra data besides (or, in addition to) the pixel data that is generated for presenting imagery on the display panel(s)of the HMD, and at least some of this extra data may be sent to the HMDas well. In some embodiments, the extra data can be packaged with the pixel data and sent to the HMDas data. At least some of the extra data packaged in the datacan be used by logic of the HMDfor purposes of presenting an image(s) corresponding to the frame on the display panel(s)of the HMD. The extra data can include, without limitation, the pose data indicative of the predicted pose of the HMD, depth data, motion vector data, parallax occlusion data, extra pixel data, and/or audio data. For example, in addition to generating the pixel data corresponding to the to-be-rendered image, the applicationmay also generate depth data (e.g., Z-buffer data) for the frame and/or extra pixel data (sometimes referred to herein as “out-of-bounds pixel data” or “additional pixel data”). Additionally, or alternatively, motion vector data based at least in part on the head tracking data received from the HMDmay be generated and transmitted to the HMDfor aiding in the processing of the pixel data on the HMD. For example, motion vector data can be generated based on a comparison of head tracking data generated at two different points in time (e.g., a comparison of head tracking data separated by a few milliseconds). Logic of the HMDcan utilize some or all of the extra data for purposes of modifying the pixel data to correct for errors in the pose prediction made ahead of time at the host computer. For example, adjustments made by the HMDmay include, without limitation, adjustments for geometric distortion, chromatic aberration, re-projection, and the like.

104 102 136 104 102 138 138 140 1 132 140 2 140 1 140 2 108 104 138 142 1 142 2 1 FIG. Before the pixel data is sent to the HMD, the host computermay implement the dual detail encoding technique disclosed herein, which allows for timely transmitting the datato the HMDunder data rate constraints on the order of 100 Mbps. As depicted in, the host computermay generate a 2D arrayof pixels for the frame with a reduced number of pixels, as compared to the number of pixels of the application-rendered frame (which may be on the order of 12.5 million pixels, as described in the example above). The 2D arrayof pixels may include first pixels representing a first copy of the frame() downscaled to a second resolution less than the first resolution at which the applicationrendered the frame, and second pixels representing a second copy of the frame() downscaled to the second resolution. “Downscale,” as used herein, means reducing the resolution, and is sometimes referred to as “down-res,” or “down-convert.” In an illustrative example, each copy of the frame(),() for an individual display panelof the HMDmay be downscaled from a first resolution of, say, 2500×2500 pixels to a second resolution of, say, 500×500 pixels. This is exemplary, and any suitable amount or degree of downscaling may be implemented. The 2D arrayof pixels may further include third pixels representing a first copy of a subregion() of the scene, and fourth pixels representing a second copy of the subregion(). The subregion may be a region of the scene for the given frame.

106 106 104 106 138 142 1 142 1 138 In some examples, the subregion may be a predetermined (or fixed) subregion. In VR, the useris looking straight forward most of the time, which means that most of the time, the useris looking at a very small fraction of the images presented on a near-eye display, such as a HMD. This fraction of the images corresponds to a central subregion of the scene, give or take 10 to 30 degrees off center. In any case, the userseldom, if ever, looks at the periphery of the scene (e.g., the corners and far edges of the imagery), and, in fact, the user's eyes may be incapable of viewing the periphery of the scene in a near-eye display other than through their peripheral vision because the angles are too extreme in a near-eye display. Accordingly, the subregion of the scene may be predetermined, in some examples, and the predetermined subregion may correspond to a central subregion of the scene. That is, in some examples, for each frame that is rendered, the 2D arrayof pixels may include pixels that represent copies of a subregion(),() of the scene that is predetermined as the central subregion of the scene, such that the subregion is the same portion of the scene for every frame and for every 2D arrayof pixels that is generated and transported, although the scenes themselves vary frame-to-frame.

120 104 102 122 102 108 104 106 108 104 136 102 106 120 102 102 104 102 106 106 108 106 106 In some examples, the subregion may be determined dynamically for individual frames. For example, the subregion may be determined for an individual frame based at least in part on eye tracking data generated by the eye tracking systemof the HMDand received by the host computerin the dataat runtime. The eye tracking data received by the host computermay be indicative of a predicted location on a display panel(s)of the HMDwhere the userwill be looking at a time when an image(s) is presented on the display panel(s)based on the encoded pixel data transmitted to the HMDin the data. In some examples, the host computermakes the prediction of where the userwill be looking when the imagery is presented. In some examples, the eye tracking systemmakes the prediction and transmits data representing the prediction to the host computer. An advantage of the host computermaking the prediction is that the prediction can be made closer in time to when the imagery is presented at the HMD, and/or the relatively higher computing capacity of the host computercan be leveraged for making the prediction. In any case, in examples where the subregion is determined dynamically (e.g., on-the-fly) frame-by-frame, the position of the subregion may change frame-to-frame such that when the user'sgaze steers to the left, the subregion is selected as a subregion of the scene that is left-of-center, or when the user'sgaze steers upward, the subregion is selected as a subregion of the scene that is above the center of the scene. In other words, eye tracking can be used to “steer” the more-detailed (e.g., higher-resolution) inner target, which is referred to herein as the “subregion” of the scene. In this way, the more-detailed subregion can be presented in imagery on the display panel(s)at the location where the useris looking so that the useris looking directly at the more-detailed (or higher-quality) portion of the imagery in the scene, and the less-detailed (or lower-quality) imagery is viewed, if at all, through the user's peripheral vision.

142 1 142 2 142 1 142 2 142 1 142 2 1 FIG. The copies of the subregion(),() may be of any suitable shape. The example ofshows the copies of the subregion(),() as being rectangular, but the copies of the subregion(),() can be of a different shape, such as elliptical, square-shaped, square-shaped with rounded corners (e.g., “squircle-shaped”), triangular, pentagonal, or any other suitable polygonal shape.

142 1 142 2 138 142 1 142 2 138 142 1 142 2 138 142 1 142 2 138 132 142 1 142 2 138 104 142 1 142 2 104 142 1 142 2 104 In some examples, the pixels of the respective copies of the subregion(),() in the 2D arraymay correspond one-to-one with original pixels of the frame that represent the subregion. In other words, the respective copies of the subregion(),() in the 2D arraymay be exact copies that have not been resampled (e.g., downscaled) in any manner. This means that each pixel in the subregions of the scene for each image that the application-rendered for a frame may be copied over as-is in the respective copies of the subregion(),() in the 2D array. In some examples, the copies of the subregions(),() in the 2D arrayhave been downscaled to a reduced resolution (i.e., a resolution that is less than the resolution at which the applicationrendered the subregions in the given frame). In these examples, the copies of the subregions(),() in the 2D arraymay be downscaled with idealized sampling, such as by downscaling the subregion in accordance with an integer ratio (e.g., 1:2). This idealized sampling for the subregion may allow for relatively high-quality resampling when the copies of the subregion are enlarged at the HMD. In other words, downscaling the copies of the subregion(),() in accordance with an integer ratio may mitigate artifacts (e.g., aliasing, blurring, etc.) in the corresponding portions of the imagery presented at the HMD, whereas downscaling the copies of the subregion(),() by 10%, 15%, or the like may cause such artifacts to be exhibited in the resulting imagery at the HMD.

142 1 142 2 132 132 132 138 108 138 108 138 138 104 132 1 FIG. The copies of the subregion(),() may be of any suitable size. A tradeoff of making the subregion as large as possible (so as to make the relatively low-quality portions of the imagery outside of the subregion less conspicuous) is that more data is required to transport the pixels for a larger subregion. Accordingly, because data rate constraints may vary by implementation, the optimal size for the subregion can vary, and there may be practical limits to increasing the size of the subregion. Furthermore, the applicationmay, in some examples, vary the resolution at which the frames are rendered frame-to-frame. In other words, the applicationmay render a first frame in a series of frames at a first resolution, and the applicationmay subsequently render a second frame (e.g., the next frame) of the series of frames at a second resolution different than the first resolution. This means that the percentage of the scene (or the image(s)) that corresponds to the subregion may vary frame-to-frame, even if the size of the subregion is fixed at M×N pixels. In the running example, the subregion may be a subregion of 800×800 pixels. Accordingly, in the example of, the 2D arrayof pixels may be an array of, say, 800×3200 pixels (or 2,560,000 pixel values for both display panels). Another example is a 2D arrayof 960×3840 pixels (or 3,686,400 pixel values for both display panels). In other words, the size of the render target for the 2D arraycan have an aspect ratio of 4:1. Thus, generating the 2D arrayof pixels results in far fewer pixels that are ultimately transmitted to the HMD, as compared to the 12,500,000 pixel values for the frame rendered by the applicationat the first resolution.

102 138 102 104 136 104 102 130 138 136 104 102 106 104 108 104 106 140 1 140 2 138 1 FIG. The host computermay encode the 2D arrayof pixels to obtain encoded pixel data for the frame, and the host computermay transmit the encoded pixel data to the HMDas part of the datathat is transmitted to the HMD, as shown in. In some examples, the host computermay utilize encoder hardware of the GPU(s)to encode the 2D arrayof pixels. Furthermore, the data, including the encoded pixel data (and possibly extra data), is transmitted to the HMDduring runtime. In this manner, the host computeris configured to encode and transport smaller images, as compared to the application-rendered images, and these smaller images include redundant data and provide varying levels of detail per eye of the userso that the HMDis able to reconstruct images that are presented on the display panelsof the HMDin such a way that the userperceives them to be high-fidelity images, even though some of the pixel data for a given frame was sacrificed (e.g., not transmitted) by downscaling the copies of the frame(),() in the 2D arrayof pixels, as described above.

104 136 104 138 104 114 102 116 104 138 At the HMD, upon receiving the data, including the encoded pixel data (and possibly extra data), the HMDmay decode the encoded pixel data to obtain the 2D arrayof pixels for the frame. In some examples, the HMDmay utilize decoder hardware of the GPU(s)to decode the encoded pixel data received from the host computer. The compositorof the HMDmay then modify the pixels of the 2D arrayin order to reconstruct the image(s) for the frame, as well as to make adjustments for geometric distortion, chromatic aberration, re-projection, and the like.

2 FIG. 1 FIG. 200 1 200 2 108 1 108 2 104 104 102 138 116 138 200 1 200 2 200 2 108 2 108 116 140 1 138 202 1 200 2 202 1 138 142 1 204 1 200 2 116 140 2 138 202 2 200 1 202 2 138 142 2 204 2 200 1 200 1 200 2 108 1 108 2 104 is a diagram illustrating images() and() that are presented on display panels() and(), respectively, of the HMD, in accordance with embodiments disclosed herein. As mentioned above, the HMDis configured to decode the encoded pixel data received from the host computerto obtain the 2D arrayof pixels for the frame shown in, and the compositormodifies the pixels of the 2D arrayin order to reconstruct the image(s)(),() for the frame. To reconstruct the right image() (e.g., for a right display panel() of a stereo pair of display panels), the compositormay enlarge the first copy of the frame() based at least in part on the corresponding pixels of the 2D arrayto obtain a first enlarged copy of the frame(), and may generate the right image() based at least in part on the first enlarged copy of the frame() and pixels of the 2D arraythat represent the first copy of a subregion() of the scene, wherein a subset of the pixels at a periphery() of the subregion are blended in the right image(). To reconstruct a left image of the two images, the compositormay enlarge the second copy of the frame() based at least in part on the corresponding pixels of the 2D arrayto obtain a second enlarged copy of the frame(), and may generate the left image() based at least in part on the second enlarged copy of the frame() and pixels of the 2D arraythat represent the second copy of the subregion(), wherein a subset of the pixels at a periphery() of the subregion are blended in the left image(). The left image() and the right image() can then be presented on respective display panels() and() of the HMD.

140 1 140 2 138 202 1 202 2 200 1 200 2 200 1 200 2 142 1 142 2 106 142 1 142 2 142 1 142 2 106 106 142 1 142 2 106 Because the copies of the frame(),() in the 2D arrayof pixels were downscaled, the enlarged copies of the frame(),() may appear blurry in the displayed images(),(). In particular, the areas of the images(),() outside of the respective subregions(),(), if looked at directly, would appear blurry, but since the user'seyes are likely to be looking directly at the subregions(),(), the relatively low-quality imagery outside of the subregions(),() is likely to go unnoticed due to those portions of the imagery being in the user'speripheral vision. By utilizing eye tracking to determine the subregion dynamically, as described above, the instances of the usernoticing the relatively low-quality imagery outside of the subregions(),() can be reduced, but even with a predetermined (or fixed) subregion, it is difficult for the average userto notice any degradation in image quality due to the dual detail encoding scheme disclosed herein.

204 1 204 2 142 1 142 2 202 1 202 2 204 1 204 2 106 142 1 142 2 200 1 200 2 204 1 204 2 142 1 142 2 200 1 200 2 142 1 142 2 202 1 202 2 142 1 142 2 200 1 200 2 202 1 202 2 142 1 142 2 204 1 204 2 200 200 142 142 200 106 200 The blended peripheries(),() render the boundary between the subregions(),() and the enlarged copies of the frame(),() inconspicuous. That is, without blending the peripheries(),() of the subregions, the usermay notice distinct boundaries (or borders) around the subregions(),() in the presented images(),(). The blending at the peripheries(),() of the subregions(),() may include fading between overlapping pixel data of the two areas of each image(),(). For example, the blending may involve interpolating between a pixel value of the subregions(),() and an overlapping pixel value of the enlarged copies of the frame(),(), where the interpolation is closer to the pixel values of the subregions(),() at points that are closer to the centers of the respective images(),(), and the interpolation gradually changes for points that are farther from the image centers to be closer to the pixel value of the enlarged copies of the frame(),() at points that are farther from the image centers. Certain shapes of the subregions(),() may result in more or less blending at the peripheries(),(). The end result of the image reconstruction for each imageis an imagewith three areas including a sharp, crisp subregion, a somewhat blurry area outside of the subregion, and an area between these two areas that is blended to make the imagelook smooth (i.e., the usercan't tell that there are three distinct areas of the image).

3 FIG.A 3 FIG.B 3 FIG.A 3 FIG.A 300 138 300 138 300 300 300 142 1 138 140 1 142 1 142 2 140 1 140 2 138 300 102 102 130 142 1 142 1 140 1 138 104 138 illustrates a first example packing layoutA for the 2D arrayof pixels described above, whileillustrates a second example packing layoutB for the 2D arrayof pixels. These are merely exemplary packing layoutsand others can be implemented. In the example of, the packing layoutA represents a vertical packing layoutA having pixels representing a first copy of the subregion() of the scene at a top of the 2D arrayof pixels, followed by pixels representing a first copy of the frame() (downscaled to a reduced resolution) below the pixels representing the first copy of the subregion(), followed by pixels representing a second copy of the subregion() below the pixels representing the first copy of the frame(), and pixels representing a second copy of the frame() (downscaled to a reduced resolution) at a bottom of the 2D arrayof pixels. This vertical packing layoutA may allow the host computerto send packets carrying some of the encoded pixel data while other pixel data is still being encoded. For example, the host computer, using the encoder of the GPU, may encode the pixels of the first copy of the subregion() and may start transmitting the encoded pixel data for the first copy of the subregion() while the pixels representing the first copy of the frame() are being encoded. This can reduce the latency of transmitting the encoded pixel data, as compared to waiting for the entire 2D arrayof pixels to be encoded before the encoded pixel data is transmitted to the HMD. Furthermore, it is to be appreciated that the ordering of the four sub-arrays of the 2D arrayshown inis exemplary. For instance, there can be as many as 24 different vertical packing layouts with 24 different ordering permutations for the four sub-arrays.

3 FIG.A 3 FIG.A 138 142 1 140 1 142 2 140 2 138 also illustrates that the dimensions of each of the four groups of pixels in the 2D arrayof pixels as being M×N pixels. In other words, the first copy of the subregion() is a rectangle with M pixels in a horizontal dimension and N pixels in a vertical dimension. As shown in, the first copy of the frame() is downscaled to a rectangle with M pixels in the horizontal dimension and N pixels in the vertical dimension. The is also the case for the second copy of the subregion() and the second copy of the frame(). In some examples, M is equal to N. In some examples, M is different than N. In an example where M is equal to N, M and N may be 500 pixels, but this is merely one example dimension that can be implemented. Accordingly, and as mentioned above, the 2D arrayof pixels may be 500 by 3200 pixels, which is far fewer pixels than the application-rendered frame.

300 138 300 140 1 138 140 2 140 1 142 1 142 2 138 142 1 142 2 300 3 FIG.B 3 FIG.A The packing layoutB shown inis another possible layout for the groups of pixels in the 2D arrayof pixels. The example packing layoutB has pixels representing the first copy of the frame() (downscaled to a reduced resolution) at a top of the 2D arrayof pixels, followed by pixels representing a second copy of the frame() (downscaled to a reduced resolution) below the pixels representing the first copy of the frame(), and the pixels representing the first and second copies of the subregion(),() are at a bottom of the 2D arrayof pixels in a side-by-side arrangement. The sub-arrays() and() in the packing layoutB may have M pixels in a horizontal dimension and N pixels in a vertical dimension, similar to that shown in.

The processes described herein are illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof (i.e., logic). In the context of software, the blocks represent computer-executable instructions that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or in parallel to implement the processes.

4 FIG. 400 100 400 illustrates a flow diagram of an example processfor implementing dual detail encoding in a distributed display system, in accordance with embodiments disclosed herein. For discussion purposes, the processis described with reference to the previous figures.

402 102 132 132 1 104 404 438 400 108 104 At, logic of the host computermay execute an application(e.g., graphics-rendering application) thereon, such as a video game(), that is tasked with rendering a frame of a series of frames for purposes of creating visual video game content to be displayed on the display device (e.g., the HMD). The following blocks-of the processmay represent sub-operations involved in rendering a frame of a series of frames, and presenting corresponding imagery on a display panel(s)of the HMD.

404 104 102 104 122 102 104 404 118 120 122 404 102 404 106 102 104 122 104 102 404 122 136 104 102 At, the HMDmay send to the host computerthat is communicatively coupled to the HMD, datathat is to be used by the host computerto render a frame. In the case of the HMD, the data sent at blockmay comprise head tracking data generated by a head tracking system, eye tracking data generated by the eye tracking system, and/or other data. In the case of another type of display device, such as a portable game device with a display, the datasent at blockmay comprise control input data (e.g., data corresponding to controls (e.g., buttons) pressed, touched, or otherwise manipulated on the game device). In the case of handheld controllers, positional controller data and/or auxiliary input data (e.g., touch sensing data, finger tracking data, etc.) may be sent to the host computerat block. Positional data of one or more handheld controllers may be forward-predicted to the location that the controller(s) would be in when the frame is displayed to the user. For an implementation where the host computeris wirelessly coupled to the HMD, the datamay be transmitted wirelessly from the HMDto the host computerat block. Although any suitable wireless communication protocol can be used, an 502.11 protocol may be used to transmit data,between the HMDand the host computer, in some examples.

406 102 122 104 122 134 102 406 At, the host computermay receive the datafrom the display device (e.g., the HMD). Although the datamay be received in various ways depending on the implementation (e.g., wirelessly, over a wired connection, via a wide-area network, etc.), the data, in at least one embodiment, may be received wirelessly via the communications interface(s)of the host computerat block.

408 132 102 108 104 102 106 132 102 104 104 108 104 At, the applicationexecuting on the host computermay render a frame for a scene at a first resolution. In some examples, the frame is rendered based on a predicted illumination time representing a time at which light emitting elements of the display panel(s)of the display device (e.g., the HMD) will illuminate for the given frame. That is, the logic of the host computermay determine a time when the photons associated with the imagery that is presented for the given frame will actually reach the user'seye(s). This predicted illumination time is a time in the future (e.g., 20-40 ms in the future) because it takes time for the applicationto generate the pixel data for the frame, and it takes time for the pixel data to be encoded and transmitted from the host computerto the HMD. It also takes time for the pixel data to be decoded, modified, and scanned out on the HMDbefore a corresponding image is ultimately presented on the display panel(s)of the display device (e.g., the HMD).

408 104 118 104 104 122 102 104 104 132 408 132 132 104 132 408 132 408 In some examples, the frame is rendered at blockbased on a predicted pose that the HMDwill be in at the predicted illumination time. For example, the head tracking systemof the HMDmay be configured to track up to six degrees of freedom of the HMD(e.g., 3D position, roll, pitch, and yaw), which can be sent in the datato the host computerto determine a predicted pose of the HMD(e.g., accounting for predicted head movement resulting in a future pose of the HMD). Accordingly, pose data indicative of the predicted pose may be provided to the applicationfor rendering the frame at block. For instance, the applicationmay call a function to receive pose data, and, in response to the function call, the applicationmay be provided with the requested pose data (predicted to the target illumination time for the given frame, and predicted based at least in part on the head tracking data received from the HMD) so that the applicationcan render the given frame according to the pose data, which corresponds to a virtual camera pose used to render the scene. The rendering at blockmay involve the applicationgenerating pixel data for the frame rendered at the first resolution (e.g., a resolution of 2.5 k, as described above). The pixel data generated at blockmay include pixel values, as described herein, for individual pixels in the array of pixels.

410 400 410 412 102 104 122 404 106 106 108 104 412 106 108 104 412 410 412 400 414 At, if a subregion of the scene is not predetermined (or fixed), the processmay follow the NO route from blockto blockwhere logic of the host computerdetermines the subregion based at least in part eye tracking data received from the display device (e.g., the HMD). This eye tracking data may have been sent in the dataat block, and it may be indicative of where the userwill be looking at the predicted illumination time for the frame. Accordingly, if the useris predicted to be looking at a center of the display panel(s)of the HMD, the subregion may be determined at blockto be a central subregion of the scene (or a center of the scene). If the useris predicted to be looking at a left side of the display panel(s)of the HMD, the subregion may be determined at blockto be a subregion that is left-of-center in the scene. If the subregion of the scene is predetermined (i.e., following the YES route from block), or after determining the subregion at block, the processmay proceed to block.

414 102 138 416 418 138 142 1 142 2 140 1 140 2 408 138 140 1 140 2 142 1 142 2 138 300 300 300 138 300 414 At, logic of the host computermay generate a 2D arrayof pixels for the frame. As shown by sub-blocksand, the 2D arrayof pixels may include two copies of the subregion(),() of the scene, and two copies of the frame(),() downscaled to a second resolution that is less than the first resolution at which the frame was rendered at block. For example, the 2D arrayof pixels may include first pixels (e.g., a first group/sub-array of pixels) representing a first copy of the frame() downscaled to the second, reduced resolution, second pixels (e.g., a second group/sub-array of pixels) representing a second copy of the frame() downscaled to the second, reduced resolution, third pixels (e.g., a third group/sub-array of pixels) representing a first copy of the subregion() of the scene, and fourth pixels (e.g., a fourth group/sub-array of pixels) representing a second copy of the subregion(). As described herein, the pixels in the 2D arraymay be arranged in any suitable packing layout, such as the packing layoutA, the packing layoutB, or any other suitable packing layout. In the vertical packing layoutA, the third pixels are at a top of the 2D array, the first pixels are below the third pixels, the fourth pixels are below the first pixels, and the second pixels are at a bottom of the 2D array of pixels, but this is merely an exemplary packing layoutA that can be implemented at block.

142 1 142 2 138 408 142 1 142 2 138 142 1 142 2 138 As noted above, the pixels of the respective copies of the subregion(),() in the 2D arraymay correspond one-to-one with original pixels of the frame rendered at block. In other words, the respective copies of the subregion(),() in the 2D arraymay be exact copies of the subregions of the rendered images for the frame, without any resampling (e.g., without downscaling) the application-rendered subregions. In examples where the copies of the subregions(),() in the 2D arrayare downscaled, the downscaling of the subregions may be performed with idealized sampling, such as by downscaling the subregion in accordance with an integer ratio (e.g., 1:2), as described above.

420 102 138 102 130 138 420 420 104 420 420 420 420 104 414 138 420 At, the logic of the host computermay encode the 2D arrayof pixels to obtain encoded pixel data for the frame. In some embodiments, encoding and/or decoding may be performed at least partially in hardware (e.g., in silicon that is built into the respective GPUs and/or CPUs of the devices). Accordingly, the host computermay utilize the encoder hardware of the GPU(s)to encode the 2D arrayof pixels at block. In some examples, the encoding performed at blockmay include compressing and/or serialize data that is to be sent to the display device (e.g., the HMD) for purposes of rendering imagery associated with the frame. In some embodiments, the encoding performed at blockinvolves changing a format of the pixel data. For example, the encoding at blockmay utilize a video encoding standard, such as High Efficiency Video Coding (HEVC) and/or extensions of HEVC, which is sometimes referred to as H.265 and/or MPEG-H Part 2. HEVC is an example of a standard that can be utilized at block, but it is to be appreciated that other suitable video encoding/compression standards can be utilized for at block, such as H.264 or VP9, for example. HEVC utilizes motion estimation for compressing pixel data, with the goal of sending less data than the uncompressed pixel data in a system where bandwidth constraints are present, yet the compressed data is sufficient to approximate the original, uncompressed data on the receiving end (e.g., at the HMD). In some examples, the operations performed at blockto generate the 2D arrayof pixels for the frame are performed as part of the encoding operation(s) at block.

420 138 130 140 1 140 2 130 142 1 142 2 142 1 142 2 140 1 140 2 102 140 1 140 2 142 1 142 2 200 1 200 2 420 200 1 200 2 In some examples, the encoding at blockmay including allocating bits to encode each group/sub-array of pixels in the 2D arrayuniformly or with variability. For example, the encoder of the GPUmay allocate fewer bits to encode the first pixels (e.g., the first group/sub-array of pixels) representing a first copy of the frame() downscaled to the second, reduced resolution, and the second pixels (e.g., the second group/sub-array of pixels) representing a second copy of the frame() downscaled to the second, reduced resolution. Meanwhile, the encoder of the GPUmay allocate more bits to encode the third pixels (e.g., the third group/sub-array of pixels) representing a first copy of the subregion() of the scene, and the fourth pixels (e.g., the fourth group/sub-array of pixels) representing a second copy of the subregion(). In other words, a first number of bits may be allocated for encoding the first pixels and the second pixels, and a second, larger number of bits may be allocated for encoding the third pixels and the fourth pixels so that more data is used to encode the copies of the subregions(),() than the copies of the downscaled frames(),(). In some examples, logic of the host computermay determine a subset of pixels representing the copies of the downscaled frames(),() that will be occluded by the pixels of the subregions(),() in the images(),(), and the encoder may, at block, allocate zero bits to the subset of pixels that will be occluded in the images(),().

422 102 104 134 422 102 422 At, the host computermay transmit (or send) to the display device (e.g., the HMD), and via the communications interface(s), the encoded pixel data. In some examples, one or more data packets carrying the encoded pixel data are transmitted wirelessly at blockvia a transceiver of the host computer. In some examples, the transmission at blockmay utilize a transmission data rate of approximately 100 Mbps.

424 104 102 124 424 104 424 At, the display device (e.g., the HMD) may receive from the host computer, and via the communications interface(s), the encoded pixel data for the frame (for the scene). In some examples, one or more data packets carrying the encoded pixel data are received wirelessly at blockvia a transceiver of the display device (e.g., the HMD). In some examples, the reception at blockmay utilize a reception data rate of approximately 100 Mbps.

426 104 138 104 114 426 426 At, logic of the display device (e.g., the HMD) may decode the encoded pixel data to obtain the 2D arrayof pixels for the frame. In some embodiments, the display device (e.g., the HMD) may utilize the decoder hardware of the GPU(s)to decode the encoded pixel data at block. In some examples, the decoding performed at blockmay include decompressing and/or deserialize data for purposes of rendering imagery associated with the frame.

428 104 116 138 430 432 428 200 1 200 2 430 140 1 140 2 202 1 202 2 432 200 1 200 2 202 1 202 2 142 1 142 2 204 1 204 2 142 1 142 2 200 2 108 2 108 116 430 140 1 138 202 1 432 200 2 202 1 138 142 1 204 1 200 2 116 430 140 2 138 202 2 200 1 202 2 138 142 2 204 2 200 1 At, the logic of the display device (e.g., the HMD), such as the compositor, may modify the pixels in the 2D arrayto obtain modified pixel data. As shown by sub-blocksand, at least some of the modifications at blockmay be for purposes of reconstructing the image(s)(),() for the frame. In general, at block, the two copies of the previously-downscaled frame(),() may be enlarged to obtain two enlarged copies of the frame(),(). At, images(),() are generated based on these enlarged copies of the frame(),() and the two copies of the subregion(),(), with blending at the peripheries(),() of the subregions(),(). For example, to reconstruct a right image() (e.g., for a right display panel() of a stereo pair of display panels), the compositormay, at block, enlarge the first copy of the frame() based at least in part on the corresponding pixels of the 2D arrayto obtain a first enlarged copy of the frame(), and may, at block, generate the right image() based at least in part on the first enlarged copy of the frame() and pixels of the 2D arraythat represent the first copy of a subregion() of the scene, wherein a subset of the pixels at a periphery() of the subregion are blended in the right image(). To reconstruct a left image of the two images, the compositormay, at block, enlarge the second copy of the frame() based at least in part on the corresponding pixels of the 2D arrayto obtain a second enlarged copy of the frame(), and may generate the left image() based at least in part on the second enlarged copy of the frame() and pixels of the 2D arraythat represent the second copy of the subregion(), wherein a subset of the pixels at a periphery() of the subregion are blended in the left image().

434 428 434 104 104 118 104 104 108 104 104 102 104 102 406 108 As shown by sub-blocks, other modifications may be performed at block, such as modifications that compensate for geometric distortion, chromatic aberration, re-projection, and the like. In an example, other modifications applied to the pixels at blockmay include re-projection adjustments that are based at least in part on an updated pose that the HMDwill be in at the illumination time for the given frame. For example, logic of the HMDmay determine, based at least in part on updated head tracking data generated by the head tracking systemof the HMD, an updated pose that the HMDwill be in at the illumination time for the given frame, which represents a time at which the light emitting elements of the display panel(s)of the HMDwill illuminate for the given frame. Because this determination is closer in time to the illumination time for the given frame, the pose prediction by the HMDmay be more accurate (e.g., have less error) than the pose prediction that was determined by the host computerfurther ahead the illumination time. Accordingly, the display device (e.g., the HMD) may compare the original predicted pose (which was based on the head tracking data received by the host computerat block) to the updated pose, which may reveal a delta (or difference) between the compared poses, and the re-projection adjustments may include rotational calculations to compensate for this delta (e.g., by shifting and/or rotating the pixel data one way or another, depending on the delta between the two pose determinations). In other words, the head tracking data is used to “fix up” the projection of the image(s) that are rendered out to the display panel(s).

436 116 104 200 1 200 2 200 1 200 2 104 108 1 108 2 108 1 108 2 2 FIG. At, the logic (e.g., the compositor) of the display device (e.g., the HMD) may output modified pixels (or modified pixel data) to a frame buffer(s) of the display device. Since the modified pixels represent the images(),() depicted in, this is also described herein as outputting the images(),() to the frame buffer(s). For a display device (e.g., a HMD) with a pair of display panels() and(), this modified pixel data may correspond to a frame that represents a pair of images to be displayed on the pair of display panels() and(), and may be output to a stereo frame buffer accordingly.

200 1 200 2 116 104 428 116 202 1 202 2 142 1 142 2 116 142 1 142 2 142 1 142 2 In some examples, the images(),() may be output to the frame buffer in a single write operation or in multiple write operations. In a single write implementation, pixel values that are output to the frame buffer are output in a single pass and the pixel values are not overwritten. This single write implementation may involve the logic (e.g., the compositor) of the display device (e.g., the HMD) performing a lookup for each pixel to determine the pixel value to output based on the pixel modifications that were made at block. In a multi-write implementation, the compositor, in a first pass, may output, to the frame buffer(s), the pixel values corresponding to the enlarged copies of the frame(),(), even those pixel values that will eventually be occluded or blended in the overlap with the subregion(),(), and then, in a second pass, the compositormay output, to the frame buffer(s), the pixel values corresponding to the copies of the subregion(),() with blending at the periphery of the subregion(),(), which may overwrite some of the previously written pixel values from the first pass. In some examples, the blending is done in a third write operation (or third pass).

438 104 200 1 200 2 108 1 108 2 104 436 108 1 108 2 104 108 1 108 2 108 1 108 2 At, logic of the display device (e.g., the HMD) may cause images(),() to be presented on respective display panels(),() of the display device (e.g., the HMD) based on the pixel values (or pixel data) output to the frame buffer at block. This may involve scanning out the pixel data to the display panels() and() of the display device (e.g., the HMD) and illuminating light emitting elements of the display panels(),() to illuminate the pixels on the display panels(),().

400 100 102 104 104 106 104 200 1 200 2 142 1 142 2 106 142 1 142 2 106 Thus, the processis an example technique for implementing a dual detail encoding scheme in a distributed display system. This dual detail encoding scheme reduces the amount of data that is to be transmitted from the host computerto a display device (e.g., the HMD) so that pixel data can be streamed in a timely manner for display of corresponding imagery on the display device (e.g., the HMD), and in such a way that userperceives the imagery as high-fidelity imagery on the display device (e.g., the HMD). For example, even though the portion of the displayed images(),() outside of the subregion(),() may be presented with relatively low quality (e.g., low resolution), the user'sgaze is likely to be directed at the subregion(),() of the scene where the image quality is relatively higher, making it seem, to the user, as though the images are of high quality.

100 104 104 104 108 1 108 2 102 104 108 1 108 2 104 100 An alternative technique for transporting pixel data in a distributed display systemunder data rate constraints is to distort the application-rendered images by enlarging the central portions of each image so that the visual information at the centers of the respective images is scaled to a larger size, encoding the distorted images to obtain encoded pixel data, and transmitting the encoded pixel data to the display device (e.g., the HMD). At the display device (e.g., the HMD), the encoded pixel data is decoded to obtain the distorted images, and the distortion is reversed at the HMDto generate the images that are presented on the display panels(),(). This alternative technique is referred to herein as a “squinch” technique. In contrast to the dual detail encoding techniques described herein, the squinch technique provides a single level of detail per eye (as opposed to two levels of detail) in that the 2D array of pixels that is generated based on the aforementioned distortion of the application-rendered images comprises, for each eye, one copy of the frame that is distorted by enlarging the central portion of the scene. Application of the squinching technique can result in a 2D array of pixels that has fewer pixels than the application-rendered frame, making it suitable for transporting pixel data under data rate constraints of, say, 100 Mbps. For example, a video game executing on the host computermay be rendering frames at a resolution of 2.5 k pixels (i.e., an image with 2500×2500 pixels) per eye, which equates to 12,500,000 pixels across two images for a HMDwith two display panels(),(), and the squinch technique described above can result in a 2D array of pixels that is on the order of 1500×1500 pixels per eye, which equates to 4,500,000 pixels that are to be transmitted to the HMDfor a given frame. Thus, the squinch solution can reduce the amount of data that is to be transported in a distributed display system, but the dual detail encoding scheme described herein can reduce the amount of data to a greater extent. Furthermore, the squinch solution may suffer from resampling artifacts on the display device, such as aliasing, blurring, or the like.

5 5 5 FIGS.A,B, andC 1 FIG. 102 102 104 106 102 106 106 104 102 104 102 106 106 illustrate alternative setups of a system for streaming data from a host computerto a display device, in accordance with embodiments disclosed herein. Briefly referring to, an example implementation is where the host computeris collocated in an environment with the display device in the form of a HMDworn by the user. For example, the host computermay be located in the user'shouse while the useris using the HMDin the house, regardless of whether the host computeris located in the same room or a different room as the HMD. Alternatively, the host computerin the form of a mobile computing device (e.g., a tablet or laptop) may be carried (e.g., in a backpack on the back of the user), thereby allowing for greater mobility. For example, the usercould be located in a public park while using such a system.

5 FIG.A 102 104 104 102 500 102 104 500 102 104 shows an alternative implementation where the host computerrepresents one or more server computers located at a geographically remote location with respect to the HMD. In this case, the HMDmay be communicatively coupled to the host computer(s)via an access point (AP), such as a wireless AP (WAP), a base station, a USB wireless dongle, etc. In an illustrative example, data is exchanged (e.g., streamed) between the host computerand the HMDvia the AP, such as by streaming data over the Internet. In this implementation, the host computerand/or the HMDmay implement one or more of the dual detail encoding techniques described herein.

5 FIG.B 5 5 FIGS.A andB 5 FIG.A 5 FIG.B 102 104 504 500 102 104 504 102 104 504 104 shows yet another alternative implementation where the host computeris communicatively coupled to the HMDvia an intermediate computing device, such as a laptop or a tablet computer. A difference betweenis that the APinmay act as a data routing device for a server computerthat is remotely located with respect to the HMD, while the intermediate computing deviceofmay act as a data routing device for a host computerthat is collocated within the same environment as the HMD. The intermediate computing devicemay even perform a portion of the rendering workload (e.g., reconstructing pixel data and/or modifying pixel data, as described herein) so that the HMDcan remain as “light-weight” as possible.

5 FIG.C 5 FIG.C 104 502 502 102 106 502 102 502 104 132 shows yet another alternative implementation where the HMDis replaced by another type of display device in the form of a portable game devicewith a display. In the setup of, the portable game devicemay send control input data to the host computeras the usermanipulates the controls of the portable game devicewith his/her hands, and the host computermay generate pixel data based at least in part on the control input data, and may transmit the encoded pixel data to the portable game device, as described herein predominantly with respect to a HMD. Accordingly, traditional game-streaming systems that are challenged by the adverse environment on most home networks may benefit from the dual detail encoding techniques and systems described herein. For example, a home personal computer running a gaming platform/applicationmay allow for continuous playback of video streams and gameplay interactions with significant data rate constraints. Use of a gaming streaming service according to the techniques and systems described herein may enable a higher quality of gameplay experience in an environment with limited bandwidth. Furthermore, while the techniques are predominantly described herein with reference to VR gaming, some or all of the techniques may apply equally to the 2D/3D gaming realm.

6 FIG. 6 FIG. 6 FIG. 104 102 104 502 104 102 104 106 106 104 106 104 106 104 104 104 104 104 104 102 104 illustrates example components of a wearable device, such as a HMD(e.g., a VR headset), and a host computer, in which the techniques disclosed herein can be implemented, according to the embodiments disclosed herein. It is to be appreciated, however, that relevant components described with respect to the HMDmay be implemented in other types of display devices (e.g., the portable game device), except that irrelevant components may be omitted from those other types of display devices. The HMDmay be implemented as a connected device that is communicatively coupled to the host computerduring operation, and/or as a standalone device. In either mode of operations, the HMDis to be worn by a user(e.g., on a head of the user). In some embodiments, the HMDmay be head-mountable, such as by allowing a userto secure the HMDon his/her head using a securing mechanism (e.g., an adjustable band) that is sized to fit around a head of a user. In some embodiments, the HMDcomprises a VR, AR, and/or MR headset that includes a near-eye or near-to-eve display(s). As such, the terms “wearable device”, “wearable electronic device”, “VR headset”, “AR headset”, “MR headset”, and “head-mounted display (HMD)” may be used interchangeably herein to refer to the deviceof. However, these types of devices are merely example of a HMD, and it is to be appreciated that the HMDmay be implemented in a variety of other form factors. It is also to be appreciated that some or all of the components shown inmay be implemented on the HMD. Accordingly, in some embodiments, a subset of the components shown as being implemented in the HMDmay be implemented on the host computeror another computing device that is separate from the HMD.

104 110 114 112 116 110 108 118 120 124 In the illustrated implementation, the HMDincludes the aforementioned processor(s), which may include one or more GPUs, as well as the memorystoring the compositorthat are executable by the processor(s), the display panel(s), the head tracking system, the eye tracking system, and the communications interface(s).

104 108 108 108 1 108 2 108 104 106 104 104 108 108 104 108 104 104 104 104 104 104 104 108 108 The HMDmay include a single display panelor multiple display panels, such as a left display panel() and a right display panel() of a stereo pair of display panels. The one or more display panelsof the HMDmay be used to present a series of image frames (herein referred to as “frames”) that are viewable by the userwearing the HMD. It is to be appreciated that the HMDmay include any number of display panels(e.g., more than two display panels, a pair of display panels, or a single display panel). Hence, the terminology “display panel,” as used in the singular herein, may refer to either display panelof a pair of display panels of a two-panel HMD, or it may refer to a single display panelof a HMDwith any number of display panels (e.g., a single-panel HMDor a multi-panel HMD). In a two-panel HMD, a stereo frame buffer may render pixels on both display panels of the HMD. In a single-panel HMD, the HMDmay include a single display paneland a pair of lenses, one for each eye to view a corresponding image displayed on a portion of the display panel.

108 104 108 108 104 The display panel(s)of the HMDmay utilize any suitable type of display technology, such as an emissive display that utilizes light emitting elements (e.g., light emitting diodes (LEDs)) or laser illumination to emit light during presentation of frames on the display panel(s). As an example, display panel(s)of the HMDmay comprise liquid crystal displays (LCDs), organic light emitting diode (OLED) displays, inorganic light emitting diode (ILED) displays, or any other suitable type of display technology for HMD applications.

108 104 108 108 The display panel(s)of the HMDmay operate at any suitable refresh rate, such as a 90 Hertz (Hz) refresh rate, a 120 Hz refresh rate, or the like, which can be a fixed refresh rate or a variable refresh rate that dynamically varies over a range of refresh rates. The “refresh rate” of a display is the number of times per second the display redraws the screen. The number of frames displayed per second may be limited by the refresh rate of the display, if using a fixed refresh rate. Thus, a series of frames may be processed (e.g., rendered) and displayed as images on the display such that a single frame of the series of frames is displayed with every screen refresh. That is, in order to present a series of images on the display panel(s), the display panel(s)may transition from frame-to-frame, in the series of frames, at the refresh rate of the display, illuminating the pixels at every screen refresh.

104 108 The display system of the HMDmay implement any suitable type of display driving scheme, such as a global flashing type of display driving scheme, a rolling band type of display driving scheme, or any other suitable type of display driving scheme. In a global flashing type of display driving scheme, the array of light emitting elements of the display illuminate simultaneously at every screen refresh, thereby flashing globally at the refresh rate. In a rolling band type of display driving scheme, individual subsets of the light emitting elements of the display can be illuminated independently and sequentially in a rolling band of illumination during an illumination time period. These types of display driving schemes may be enabled by the light emitting elements being individually addressable. If the array of pixels and the array of light emitting elements on the display panel(s)are arranged in rows and columns (but not necessarily with a one-pixel per one-light emitting element correspondence), individual rows and/or individual columns of light emitting elements may be addressed in sequence, and/or individual groups of contiguous rows and/or individual groups of contiguous columns of light emitting elements may be addressed in sequence for a rolling band type of display driving scheme.

108 104 108 108 In general, as used herein, “illuminating a pixel” means illuminating the light emitting element that corresponds to that pixel. For example, a LCD illuminates a light emitting element of a backlight to illuminate the corresponding pixel(s) of the display. Furthermore, as used herein, a “subset of pixels” may comprise an individual pixel or multiple pixels (e.g., a group of pixels). In order to drive the display panel(s), the HMDmay include, among other things, a display controller(s), display driver circuitry, and similar electronics for driving the display panel(s). Display driver circuitry may be coupled to the array of light emitting elements of the display panel(s)via conductive paths, such as metal traces, on a flexible printed circuit. In an example, a display controller(s) may be communicatively coupled to the display driver circuitry and configured to provide signals, information, and/or data to the display driver circuitry. The signals, information, and/or data received by the display driver circuitry may cause the display driver circuitry to illuminate the light emitting elements in a particular way. That is, the display controller(s) may determine which light emitting element(s) is/are to be illuminated, when the element(s) is/are to illuminate, and the level of light output that is to be emitted by the light emitting element(s), and may communicate the appropriate signals, information, and/or data to the display driver circuitry in order to accomplish that objective.

112 110 600 104 104 602 112 104 602 104 104 602 The computer-readable mediamay store additional functional modules that are executable on the processor(s), although the same functionality may alternatively be implemented in hardware, firmware, or as a SOC, and/or other logic. For example, an operating system modulemay be configured to manage hardware within and coupled to the HMDfor the benefit of other modules. In addition, in some instances the HMDmay include one or more applicationsstored in the memoryor otherwise accessible to the HMD. For example, the application(s)may include, without limitation, a video game application (e.g., a basic video game with graphics that are less-computationally-intensive to process), a video playback application (e.g., an application that accesses a video content library stored on the HMDand/or in the cloud), etc. The HMDmay include any number or type of applicationsand is not limited to the specific examples described herein.

104 604 606 604 604 604 106 604 104 104 604 104 604 Generally, the HMDhas input devicesand output devices. The input devicesmay include control buttons. In some implementations, one or more microphones may function as input devicesto receive audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors (e.g., inertial measurement unit (IMU)) may function as input devicesto receive gestural input, such as a hand and/or head motion of the user. In some embodiments, additional input devicesmay be provided in the form of a keyboard, keypad, mouse, touch screen, joystick, and the like. In other embodiments, the HMDmay omit a keyboard, keypad, or other similar forms of mechanical input. Instead, the HMDmay be implemented relatively simplistic forms of input device, a network interface (wireless or wire-based), power, and processing/memory capabilities. For example, a limited set of one or more input components may be employed (e.g., a dedicated button to initiate a configuration, power on/off, etc.) so that the HMDcan thereafter be used. In one implementation, the input device(s)may include control mechanisms, such as basic volume control button(s) for increasing/decreasing volume, as well as power and reset buttons.

606 108 108 108 606 The output devicesmay include a display panel(s), which may include one or multiple display panels(e.g., a stereo pair of display panels), as described herein. The output devicesmay further include, without limitation, a light element (e.g., LED), a vibrator to create haptic sensations, a speaker(s) (e.g., headphones), and/or the like. There may also be a simple light element (e.g., LED) to indicate a state such as, for example, when power is on.

104 124 910 102 124 124 104 102 The HMDmay further include the communications interface(s)including, without limitation, one or more antennas(E.g., of a transceiver) to facilitate a wireless connection to a network and/or to a second device, such as the host computer, as described herein. The communications interface(s)may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), and so on. It is to be appreciated that the communications interface(s)of the HMDmay further include physical ports to facilitate a wired connection to a network and/or a second device, such as the host computer.

104 612 108 612 612 612 108 612 108 612 108 The HMDmay further include optical subsystemthat directs light from the display panel(s)to a user's eye(s) using one or more optical elements. The optical subsystemmay include various types and combinations of different optical elements, including, without limitations, such as apertures, lenses (e.g., Fresnel lenses, convex lenses, concave lenses, etc.), filters, and so forth. In some embodiments, one or more optical elements in optical subsystemmay have one or more coatings, such as anti-reflective coatings. Magnification of the image light by optical subsystemallows display panel(s)to be physically smaller, weigh less, and consume less power than larger displays. Additionally, magnification of the image light may increase a field of view (FOV) of the displayed content (e.g., images). For example, the FOV of the displayed content is such that the displayed content is presented using almost all (e.g., 120-150 degrees diagonal), and in some cases all, of the user's FOV. AR applications may have a narrower FOV (e.g., about 40 degrees FOV). Optical subsystemmay be designed to correct one or more optical errors, such as, without limitation, barrel distortion, pincushion distortion, longitudinal chromatic aberration, transverse chromatic aberration, spherical aberration, comatic aberration, field curvature, astigmatism, and so forth. In some embodiments, content provided to display panel(s)for display is pre-distorted (e.g., by the applied geometric distortion adjustments and/or chromatic aberration adjustments described herein), and optical subsystemcorrects the distortion when it receives image light from display panel(s)generated based on the content.

104 614 614 614 104 104 The HMDmay further include one or more sensors, such as sensors used to generate motion, position, and orientation data. These sensorsmay be or include gyroscopes, accelerometers, magnetometers, video cameras, color sensors, or other motion, position, and orientation sensors. The sensorsmay also include sub-portions of sensors, such as a series of active or passive markers that may be viewed externally by a camera or color sensor in order to generate motion, position, and orientation data. For example, a VR headset may include, on its exterior, multiple markers, such as reflectors or lights (e.g., infrared or visible light) that, when viewed by an external camera or illuminated by a light (e.g., infrared or visible light), may provide one or more points of reference for interpretation by software in order to generate motion, position, and orientation data. The HMDmay include light sensors that are sensitive to light (e.g., infrared or visible light) that is projected or broadcast by base stations in the environment of the HMD.

614 616 616 616 616 104 104 616 104 616 104 104 104 616 616 614 102 102 104 In an example, the sensor(s)may include an inertial measurement unit (IMU). IMUmay be an electronic device that generates calibration data based on measurement signals received from accelerometers, gyroscopes, magnetometers, and/or other sensors suitable for detecting motion, correcting error associated with IMU, or some combination thereof. Based on the measurement signals such motion-based sensors, such as the IMU, may generate calibration data indicating an estimated position of HMDrelative to an initial position of HMD. For example, multiple accelerometers may measure translational motion (forward/back, up/down, left/right) and multiple gyroscopes may measure rotational motion (e.g., pitch, yaw, and roll). IMUcan, for example, rapidly sample the measurement signals and calculate the estimated position of HMDfrom the sampled data. For example, IMUmay integrate measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on HMD. The reference point is a point that may be used to describe the position of the HMD. While the reference point may generally be defined as a point in space, in various embodiments, reference point is defined as a point within HMD(e.g., a center of the IMU). Alternatively, IMUprovides the sampled measurement signals to an external console (or other computing device), which determines the calibration data. The sensorsmay include sensors of one or more handheld controllers that are part of a HMD system. Accordingly, in some embodiments, the controllers may transmit sensor data to the host computerand the host computermay fuse the sensor data it receives from the HMDand the handheld controller(s).

614 The sensorsmay operate at relatively high frequencies in order to provide sensor data at a high rate. For example, sensor data may be generated at a rate of 1000 Hz (or 1 sensor reading every 1 millisecond). In this way, one thousand readings are taken per second. When sensors generate this much data at this rate (or at a greater rate), the data set used for predicting motion is quite large, even over relatively short time periods on the order of the tens of milliseconds.

614 104 104 614 As mentioned, in some embodiments, the sensorsmay include light sensors that are sensitive to light emitted by base stations in the environment of the HMDfor purposes of tracking position and/or orientation, pose, etc., of the HMDin 3D space. The calculation of position and/or orientation may be based on timing characteristics of light pulses and the presence or absence of light detected by the sensors.

104 118 118 614 106 118 104 132 118 104 132 106 108 118 104 104 104 The HMDmay further include the aforementioned head tracking system. The head tracking systemmay leverage one or more of the sensorto track head motion, including head rotation, of the user, as described above. For example, the head tracking systemcan track up to six degrees of freedom of the HMD(i.e., 3D position, roll, pitch, and yaw). These calculations can be made at every frame of a series of frames so that the applicationcan determine how to render a scene in the next frame in accordance with the head position and orientation. In some embodiments, the head tracking systemis configured to generate head tracking data that is usable to predict a future pose (position and/or orientation) of the HMDbased on current and/or past data, and/or based on the known/implied scan out latency of the individual subsets of pixels in a display system. This is because the applicationis asked to render a frame before the useractually sees the light (and, hence, the image) on the display panel(s). Accordingly, a next frame can be rendered based on this future prediction of head position and/or orientation that was made at an earlier point in time. Rotation data provided by the head tracking systemcan be used to determine both direction of HMDrotation, and amount of HMDrotation in any suitable unit of measurement. For example, rotational direction may be simplified and output in terms of positive or negative horizontal and positive or negative vertical directions, which correspond to left, right, up, and down. Amount of rotation may be in terms of degrees, radians, etc. Angular velocity may be calculated to determine a rate of rotation of the HMD.

104 120 120 104 120 104 104 120 106 120 120 106 108 106 108 120 106 108 106 104 106 108 104 The HMDmay further include the aforementioned eye tracking systemthat generates eye tracking data. The eye tracking systemmay include, without limitation, a camera or other optical sensor inside HMDto capture image data (or information) of a user's eyes, and the eye tracking systemmay use the captured data/information to determine motion vectors, interpupillary distance, interocular distance, a 3D position of each eye relative to HMD, including a magnitude of torsion and rotation (i.e., roll, pitch, and yaw) and gaze directions for each eye. In one example, infrared light is emitted within HMDand reflected from each eye. The reflected light is received or detected by a camera of the eye tracking systemand analyzed to extract eye rotation from changes in the infrared light reflected by each eye. Many methods for tracking the eyes of a usercan be used by eye tracking system. Accordingly, eye tracking systemmay track up to six degrees of freedom of each eye (i.e., 3D position, roll, pitch, and yaw) and at least a subset of the tracked quantities may be combined from two eyes of a userto estimate a gaze point (i.e., a 3D location or position in the virtual scene where the user is looking), which may map to a location(s) on the display panel(s)for predicting where the userwill be looking in terms of an individual subset (e.g., a row) or a group of contiguous subsets (e.g., a group of contiguous rows) of the pixels of the display panel(s). For example, eye tracking systemmay integrate information from past measurements, measurements identifying a position of a user'shead, and 3D information describing a scene presented by display panel(s). Thus, information for the position and orientation of the user'seyes is used to determine the gaze point in a virtual scene presented by HMDwhere the useris looking, and to map that gaze point to a location(s) on the display panel(s)of the HMD.

102 126 130 128 132 134 In the illustrated implementation, the host computerincludes the aforementioned processor(s), which may include one or more GPUs, as well as the memorystoring the application(s), and the communications interface(s).

128 618 102 128 620 132 1 622 622 620 106 622 620 132 1 132 1 620 The memorymay further store an operating system modulethat is configured to manage hardware within and coupled to the host computerfor the benefit of other modules. The memorymay further store a video game clientthat is configured to execute one or more video games, such as the video game(), within a video game library. The video games in the video game librarymay be retrieved and executed by loading the video game client. In an example, a usermay choose to play one of multiple video games they have purchased and downloaded to the video game libraryby loading the video game clientand selecting a video game() to start execution of the video game(). The video game clientmay allow users to login to a video game service using credentials (e.g., a user account, password, etc.).

102 134 624 104 134 134 102 104 502 The host computermay further include the communications interface(s)including, without limitation, one or more antennas(e.g., of a transceiver) to facilitate a wireless connection to a network and/or to a second device, such as the HMD. The communications interface(s)may implement one or more of various wireless technologies, such as Wi-Fi, Bluetooth, radio frequency (RF), and so on. It is to be appreciated that the communications interface(s)of the host computermay further include physical ports to facilitate a wired connection to a network and/or a second device, such as the HMDor another type of display device, such as the portable game device.

102 626 628 626 628 Generally, the host computerhas input devicesand output devices. The input devicesmay a keyboard, keypad, mouse, touch screen, joystick, control buttons, microphones, cameras, and the like. The output devicesmay include, without limitation, a display(s), a light element (e.g., LED), a vibrator to create haptic sensations, a speaker(s) (e.g., headphones), and/or the like.

Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as illustrative forms of implementing the claims.