Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A method comprising: capturing, using a video capture device associated with a computing device, a video stream of a physical activity scene, the video stream including an activity object, the activity object including a visualization marker, a reference object depicting a type of a tangible interface object to form, and an interaction area, the interaction area including the tangible interface object formed in the interaction area based on the type of the tangible interface object depicted by the reference object; identifying, using a processor of the computing device, the activity object based on the visualization marker; determining, using the processor of the computing device, a virtual object based on the identity of the activity object; displaying, on a display of the computing device, a graphical user interface embodying a virtual scene, the virtual scene including the virtual object; detecting, using the processor of the computing device, a formation of the tangible interface object in the interaction area by: detecting a line segment of the tangible interface object; matching the line segment to a reference page that includes a depiction of the reference object; and identifying a boundary of the tangible interface object based on the matching; generating, using the processor of the computing device, a virtualization based on the formation of the tangible interface object; and updating, on the display of the computing device, the graphical user interface to include the virtualization.
2. The method of claim 1 , wherein the tangible interface object is a drawing and the formation of the tangible interface object is a creation of the drawing by a user.
3. The method of claim 1 , wherein the reference object provides an indicator of the formation of the tangible interface object.
4. The method of claim 1 , wherein detecting the formation of the tangible interface object in the interaction area further comprises: extracting, using the processor of the computing device, an image of the tangible interface object from the video stream based on the boundary.
5. The method of claim 4 , wherein updating the graphical user interface to include the virtualization further comprises: matching the boundary of the extracted image with a portion of the virtual object.
This invention relates to virtual object integration in graphical user interfaces, specifically addressing the challenge of seamlessly incorporating virtual objects into real-world environments displayed in augmented reality (AR) or virtual reality (VR) systems. The method involves extracting an image from a captured scene, identifying a boundary within that image, and updating the graphical user interface to include a virtual object. The key innovation is in matching the boundary of the extracted image with a portion of the virtual object to ensure proper alignment and integration. This process may involve analyzing the extracted image to determine the boundary's shape, size, or orientation, then adjusting the virtual object's position, scale, or rotation to align with the boundary. The method may also include dynamically updating the virtual object's appearance or behavior based on changes in the captured scene or user interactions. The goal is to create a cohesive and realistic visual experience by ensuring the virtual object appears naturally integrated with the real-world environment. This technique is particularly useful in AR applications where virtual elements must interact with physical surroundings in a visually convincing manner.
6. The method of claim 1 , wherein the visualization marker represents a type of the activity object and wherein identifying the activity object based on the visualization marker further comprises: identifying, using the processor of the computing device, the type of the activity object based on the visualization marker.
7. The method of claim 1 , wherein the interaction area includes an outline of the tangible interface object.
8. A physical activity scene visualization system comprising: a video capture device coupled for communication with a computing device, the video capture device being adapted to capture a video stream that includes an activity object, the activity object including a reference object depicting a type of a tangible interface object to form and an interaction area, the interaction area including the tangible interface object formed in the interaction area based on the type of the tangible interface object depicted by the reference object; a detector coupled to the computing device, the detector being adapted to identify within the video stream the activity object and a formation of the tangible interface object in the interaction area by detecting a line segment of the tangible interface object, matching the line segment to a reference page that includes a depiction of the reference object, and identifying a boundary of the tangible interface object based on the matching; a processor of the computing device, the processor being adapted to identify the activity object, determine a virtual object based on the identity of the activity object, and generate a virtualization based on the formation of the tangible interface object; and a display coupled to the computing device, the display being adapted to display a graphical user interface embodying a virtual scene, the virtual scene including the virtual object and updating the graphical user interface to include the virtualization.
9. The physical activity scene visualization system of claim 8 , wherein the tangible interface object is a drawing and the formation of the tangible interface object is a creation of the drawing by a user.
10. The physical activity scene visualization system of claim 8 , wherein the reference object provides an indicator of the formation of the tangible interface object.
The system relates to visualizing physical activity scenes, particularly for enhancing user interaction with digital interfaces. The problem addressed is the lack of intuitive feedback when users manipulate tangible objects in a physical space to control digital interfaces, making it difficult to understand how actions translate into digital effects. The system includes a tangible interface object that users can physically manipulate, such as moving or rotating, to interact with a digital interface. A reference object is used to provide visual feedback, indicating how the tangible object's movements form or modify the digital interface. For example, the reference object may display alignment markers, color changes, or other visual cues to show when the tangible object is correctly positioned to create or adjust a digital element. This feedback helps users understand the relationship between their physical actions and the resulting digital changes, improving usability and interaction accuracy. The system may also include sensors to track the tangible object's position and orientation, ensuring precise control over the digital interface. The reference object dynamically updates based on real-time input, providing continuous feedback to guide the user. This approach enhances the intuitive nature of tangible user interfaces, making them more accessible and effective for various applications, such as gaming, design, or industrial control systems.
11. The physical activity scene visualization system of claim 8 , wherein the detector is further configured to: extract an image of the tangible interface object from the video stream based on the boundary.
12. The physical activity scene visualization system of claim 11 , wherein the processor configured to generate the virtualization is further configured to match the boundary of the extracted image with a portion of the virtual object.
The system relates to visualizing physical activity scenes by integrating real-world imagery with virtual objects. The problem addressed is the difficulty in accurately aligning extracted real-world images with virtual elements to create a cohesive and realistic visualization. The system extracts an image from a physical activity scene, such as a sports event or fitness activity, and processes it to identify key features. A virtual object, such as a graphical overlay or augmented reality element, is then generated and aligned with the extracted image. The alignment process ensures that the virtual object's boundary matches a portion of the extracted image, enhancing the realism and accuracy of the visualization. This alignment may involve scaling, rotating, or positioning the virtual object to fit seamlessly with the real-world image. The system may also adjust lighting, shadows, or other visual effects to further integrate the virtual object with the physical scene. The goal is to provide a visually coherent and immersive experience for users, such as athletes, coaches, or spectators, by combining real-world data with virtual enhancements. The system may be used in applications like sports analysis, training simulations, or interactive fitness tracking.
13. The physical activity scene visualization system of claim 8 , wherein the activity object includes a visualization marker and the visualization marker represents a type of the activity object and the detector is further configured to: identify the type of the activity object based on the visualization marker.
14. The physical activity scene visualization system of claim 8 , wherein the interaction area includes an outline of the tangible interface object.
15. A method comprising: capturing, using a video capture device associated with a computing device, a video stream of a physical activity scene, the video stream including an activity object, the activity object including a visualization marker, a reference object depicting a type of a drawing to form, and an interaction area, the interaction area including the drawing by a user in the interaction area based on the type of the drawing depicted by the reference object; identifying, using a processor of the computing device, a type of the activity object based on the visualization marker; determining, using the processor of the computing device, a virtual scene for display, the virtual scene including a virtual object to prompt the user; displaying, on a display of the computing device, a graphical user interface embodying a virtual scene, the virtual scene including the virtual object; detecting, using the processor of the computing device, the drawing in the interaction area by: detecting a line segment of the drawing; matching the line segment to a reference page that includes a depiction of the reference object; and identifying a boundary of the drawing based on the matching; generating, using the processor of the computing device, a virtual image of the drawing by the user; and updating, on the display of the computing device, the graphical user interface to include the virtual image of the drawing.
This invention relates to interactive digital drawing systems that combine physical and virtual elements to guide users in creating drawings. The system addresses the challenge of providing structured, guided drawing experiences by integrating physical reference objects with virtual feedback. A video capture device captures a video stream of a physical activity scene, which includes an activity object. The activity object contains a visualization marker, a reference object depicting a type of drawing to form, and an interaction area where the user draws. The system identifies the type of activity object using the visualization marker and determines a corresponding virtual scene for display. The virtual scene includes a virtual object that prompts the user. The system detects the user's drawing in the interaction area by analyzing line segments, matching them to a reference page depicting the reference object, and identifying the drawing's boundary. A virtual image of the drawing is generated and displayed alongside the virtual scene, providing real-time feedback. This approach enhances user engagement by blending physical drawing with digital guidance and visualization.
16. The method of claim 15 , wherein the activity object is a drawing pad and the visualization marker represents a page of the drawing pad.
17. The method of claim 15 , wherein the virtual object is a graphical representation of a portion of a graphical representation that is missing a feature and the drawing fills in that portion of the graphical representation with the missing feature.
18. The method of claim 17 , wherein updating the graphical user interface to include the virtual image of the drawing further comprises: matching, using the processor of the computing device, the boundary of the drawing to the portion of the graphical representation with the missing feature.
19. The method of claim 15 , wherein generating the virtual image further comprises: extracting, using the processor of the computing device, an image of the drawing from the video stream based on the boundary.
20. The method of claim 15 , wherein the interaction area includes an outline to assist the user in creating the drawing.
This invention relates to digital drawing tools designed to improve user interaction and precision in creating drawings. The problem addressed is the difficulty users face in accurately placing and manipulating drawing elements within a digital workspace, particularly when freehand drawing or positioning objects. The invention provides a method for enhancing user interaction by incorporating an outline within the interaction area of a digital drawing tool. This outline serves as a visual guide, helping users align and position drawing elements more precisely. The interaction area is a designated region within the digital workspace where user inputs, such as touch or stylus interactions, are processed to generate or modify drawings. The outline can be dynamic, adjusting in real-time based on user input or predefined constraints, ensuring flexibility and adaptability. The method may also include additional features, such as snapping to grid lines or predefined shapes, to further assist in accurate drawing. The outline can be customized in appearance, such as color, thickness, or transparency, to suit user preferences or specific drawing tasks. This approach improves usability and efficiency, making digital drawing more intuitive and precise for both novice and experienced users.
Unknown
April 13, 2021
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.