An apparatus and system to provide QoE metrics reporting mechanisms for RTP-based 360-degree video delivery in live immersive streaming and real-time immersive conversational service applications are described for both in-camera and network-based stitching. Initial and desired parameters for viewports used in a teleconference are exchanged, and the teleconference established using 360° media. RTP FoV reports sent during the teleconference each contain viewport orientation information, as well as information for the QoE metrics.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
6. The apparatus of claim 2, wherein the renderer is configured to use decoder metadata information contained in Supplemental Information Enhancement (SEI) messages contained in the elementary stream to render video and audio signals of the 3D video with a pose and a knowledge of a horizontal and vertical field of view to determine the viewport.
This invention relates to rendering 3D video content with enhanced viewport determination using metadata from Supplemental Information Enhancement (SEI) messages in an elementary stream. The technology addresses the challenge of accurately rendering 3D video and audio signals by leveraging decoder metadata to optimize the viewing experience. The apparatus includes a renderer that processes SEI messages containing metadata, which provides information about the pose (orientation and position) of the viewer or display device, as well as the horizontal and vertical field of view. This metadata enables the renderer to dynamically determine the optimal viewport for displaying the 3D content, ensuring proper alignment and perspective. The system may also include a decoder that extracts the SEI messages from the elementary stream and provides the metadata to the renderer. The renderer then uses this information to adjust the rendering of video and audio signals, ensuring that the 3D content is displayed with the correct spatial orientation and field of view. This approach improves the accuracy and immersion of 3D video playback by dynamically adapting to the viewer's perspective and display capabilities.
7. The apparatus of claim 6, wherein the renderer is configured to use the decoder metadata information for region-wise unpacking, projection de-mapping and rotation toward creating spherical content for each eye of a user. The VR Renderer uses the decoded signals and rendering metadata.
This invention relates to virtual reality (VR) rendering systems that process and display spherical content for immersive viewing. The technology addresses the challenge of efficiently generating high-quality VR content by leveraging decoder metadata to optimize rendering for each eye of the user. The system includes a renderer that utilizes decoder metadata to perform region-wise unpacking, projection de-mapping, and rotation of the content. This ensures accurate and efficient transformation of the decoded signals into spherical content tailored for stereoscopic viewing. The renderer also incorporates rendering metadata to further refine the display output, enhancing visual fidelity and user experience. The apparatus is designed to handle the complex spatial transformations required for VR, ensuring seamless and distortion-free presentation of 360-degree content. By dynamically adjusting the rendering process based on metadata, the system improves processing efficiency and reduces computational overhead while maintaining high-quality visual output. This approach is particularly useful in VR applications where real-time performance and low latency are critical.
8. The apparatus of claim 2, wherein the processing circuitry is configured to control the RTP receiver, the HVEC decoder, the texture-to-sphere mapper, and the renderer based on media control information, dynamic user pose, and display and device capabilities using a display resolution, a maximum display refresh rate, a field of view in both horizontal and vertical directions, an eye-to-screen distance, a lens separation distance and operating system support.
This invention relates to a system for rendering immersive media content, such as virtual reality (VR) or augmented reality (AR) experiences, with optimized performance based on user and device parameters. The system addresses the challenge of efficiently processing and displaying high-resolution, low-latency immersive content by dynamically adjusting rendering parameters to match user pose, display capabilities, and device constraints. The apparatus includes processing circuitry that controls an RTP (Real-time Transport Protocol) receiver, an HEVC (High Efficiency Video Coding) decoder, a texture-to-sphere mapper, and a renderer. The RTP receiver handles real-time media streaming, while the HEVC decoder decompresses encoded video data. The texture-to-sphere mapper converts decoded video frames into a spherical format suitable for immersive displays, and the renderer generates the final output for presentation. The processing circuitry dynamically adjusts these components based on media control information, user pose (e.g., head movements), and device capabilities. Key parameters include display resolution, maximum refresh rate, field of view (horizontal and vertical), eye-to-screen distance, lens separation distance, and operating system support. By optimizing these factors, the system ensures smooth, high-quality rendering while minimizing latency and computational overhead. This approach enhances user experience in VR/AR applications by adapting to real-time changes in user interaction and device limitations.
9. The apparatus of claim 1, wherein the processing circuitry is further configured to negotiate Vital Product Data (VPD) capability, and the viewport switching latency metric includes latency and quality-related factors when viewport movement causes quality degradations, the latency and quality-related factors including viewport quality within a viewport.
This invention relates to a system for optimizing viewport switching in a video streaming or rendering environment, particularly addressing latency and quality degradation issues during viewport movement. The apparatus includes processing circuitry that negotiates Vital Product Data (VPD) capability, which likely involves exchanging metadata or configuration details between devices to ensure compatibility and performance. The system measures viewport switching latency, incorporating both latency and quality-related factors when viewport changes cause visual degradation. These factors include the quality of the viewport itself, ensuring that the displayed content remains visually acceptable despite dynamic adjustments. The apparatus dynamically assesses these metrics to improve user experience by minimizing disruptions during viewport transitions, such as in virtual reality (VR), augmented reality (AR), or other immersive applications. The negotiation of VPD capability ensures that the system can adapt to different devices or network conditions, maintaining optimal performance. The invention aims to provide seamless, high-quality viewport switching by balancing latency and visual fidelity, addressing challenges in real-time rendering and streaming environments.
10. The apparatus of claim 9, wherein the viewport quality is represented by a quality ranking (QR) value and a pixel resolution of one or more regions within the viewport.
A system for evaluating and optimizing viewport quality in digital displays or imaging devices addresses the challenge of assessing and improving the visual fidelity of displayed content. The system determines viewport quality by analyzing a quality ranking (QR) value and the pixel resolution of one or more regions within the viewport. The QR value quantifies the overall visual quality, while the pixel resolution specifies the detail level in different areas. This approach allows for targeted adjustments to enhance display performance, such as adjusting resolution, contrast, or other visual parameters based on the QR value and resolution data. The system may also incorporate user preferences or environmental factors to refine quality assessments. By dynamically evaluating and adapting viewport quality, the system ensures optimal viewing experiences across various devices and conditions. This technology is particularly useful in applications like virtual reality, augmented reality, high-resolution displays, and medical imaging, where visual accuracy and clarity are critical. The system may integrate with existing display technologies or operate as a standalone quality assessment module.
11. The apparatus of claim 10, wherein the processing circuitry is further configured to determine whether the viewport has a comparable quality to another viewport, in response to more than one quality ranking region being visible inside the viewport, aggregated viewport quality factors are calculated as an area-weighted average QR and an area-weighted pixel resolution.
This invention relates to a system for evaluating the visual quality of viewports in a display environment, particularly where multiple quality-ranking regions are visible within a single viewport. The problem addressed is the need to assess and compare the overall quality of a viewport when it contains overlapping or adjacent regions with different quality rankings, ensuring accurate quality determination for display optimization. The apparatus includes processing circuitry that calculates quality factors for a viewport by analyzing visible regions within it. When multiple quality-ranking regions are present, the system computes aggregated viewport quality factors. These factors include an area-weighted average quality ranking (QR) and an area-weighted pixel resolution. The area-weighted average QR is derived by considering the proportion of each quality-ranking region's area within the viewport, while the area-weighted pixel resolution accounts for the resolution distribution across these regions. This approach ensures that the viewport's overall quality is accurately represented, even when different regions contribute differently to the visual experience. The system can then determine whether the viewport's quality is comparable to another viewport based on these aggregated factors, enabling informed decisions for display adjustments or content rendering. This method is particularly useful in applications requiring dynamic quality assessment, such as virtual reality, augmented reality, or multi-region display systems.
12. The apparatus of claim 11, wherein in response to movement of the viewport so that the viewport includes at least one new quality ranking region, the processing circuitry is configured to start a switch event.
This invention relates to a system for dynamically adjusting content display based on user interaction with a viewport, particularly in applications where content is ranked by quality. The problem addressed is the need to efficiently update displayed content when a user navigates to new regions of a ranked content space, ensuring seamless and relevant content presentation. The apparatus includes processing circuitry that monitors viewport movement and detects when the viewport enters a new quality ranking region. Quality ranking regions are predefined areas within a content space where content is grouped based on its quality metrics, such as relevance, importance, or user engagement. When the viewport moves to include at least one new quality ranking region, the processing circuitry initiates a switch event. This event triggers the system to update the displayed content, ensuring that the viewport now shows content from the newly entered quality ranking region. The update may involve fetching new content, reordering existing content, or adjusting display parameters to reflect the quality ranking of the new region. The system ensures that content displayed in the viewport remains relevant to the user's current focus area, improving user experience and content accessibility. The apparatus may be part of a larger content management system, such as a search engine, media player, or data visualization tool, where dynamic content updates are essential for maintaining context and relevance.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
July 29, 2021
May 21, 2024
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.