One embodiment of the present invention sets forth a technique for providing an end user with a digital pen embedded with a spatially-aware miniature projector for use in a design environment. Paper documents are augmented to allow a user to access additional information and computational tools through projected interfaces. Virtual ink may be managed in single and multi-user environments to enhance collaboration and data management. The spatially-aware projector pen provides end-users with dynamic visual feedback and improved interaction capabilities.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A spatially-aware projection pen system, comprising: an input mechanism configured to be activated and deactivated by a user; a position tracking mechanism configured to determine a position of the spatially-aware projection pen within a three-dimensional space and relative to a display surface; a memory configured to store design data, image data, and an application program, wherein the design data comprises computer-aided design (CAD) data; a processor configured to: execute the application program to generate at least a portion of the image data based on the CAD data and printed content on the display surface, receive input signals from the input mechanism and the position tracking mechanism, and generate a projected image based on the image data, the input signals and the position of the spatially-aware projection pen, wherein the proiected image includes a plurality of visual components, and wherein a geometry and a location of each visual component corresponds to a different content portion of the printed content; a projector configured to display the projected image overlaid on the printed content, wherein a location of the projected image is locked relative to the printed content and remains static as the position of the spatially-aware projection pen changes; and an enclosure configured to be grasped by a human hand and house the input mechanism, the position tracking mechanism, the memory, the projector, and the processor.
A spatially-aware projection pen system comprises a pen-shaped device that can project images onto a surface, augmenting printed content with interactive digital information. The pen includes a button for user activation, a position tracking system to determine its location in 3D space relative to the display surface, a memory to store CAD data, image data, and an application. A processor executes the application to generate image data based on CAD data and printed content. It uses input from the activation button and position tracking to create a projected image containing visual components aligned with different parts of the printed content. A projector displays this image overlaid on the printed content, locked in place relative to the printed content, even as the pen moves. All components are housed within a hand-held enclosure.
2. The system of claim 1 , further comprising a force sensor that is configured to detect a force applied to a tip of the spatially-aware projection pen and provide a force signal to the processor indicating whether the tip of the spatially-aware projection pen is in contact with the display surface.
The spatially-aware projection pen system described in claim 1 also contains a force sensor in the pen tip to detect contact with the display surface. This sensor sends a force signal to the processor, indicating whether the pen tip is touching the surface. This allows the system to know when the user is interacting directly with the surface.
3. The system of claim 2 , wherein the force signal provided by the force sensor is captured as virtual ink by the processor and stored in the memory as the image data representing pen strokes input by the user.
Building on the system described in claim 2, the force sensor signal (indicating pen tip contact) is used to create "virtual ink." When the force sensor detects contact, the processor captures and stores pen strokes as image data, effectively simulating digital ink. These strokes are then saved in memory as part of the image data.
4. The system of claim 2 , wherein the projector is further configured to display virtual ink corresponding to pen strokes in the projected image when the tip of the spatially-aware projection pen is in contact with the display surface.
Expanding on the system described in claim 2, the projector displays virtual ink corresponding to the pen strokes in the projected image. This means when the force sensor detects that the pen tip is in contact with the display surface, the projector shows a visual representation of ink being drawn in real-time.
5. The system of claim 1 , wherein the processor is configured to render a frame of pixel data using the design data and based on the position of the spatially-aware projection pen.
In the spatially-aware projection pen system described in claim 1, the processor renders a frame of pixel data based on design data (CAD data) and the current position of the pen. This enables the system to create and display images that change depending on where the pen is located relative to the display surface.
6. The system of claim 5 , wherein the processor is further configured to store the frame of pixel data in the memory as image data.
Continuing from the system described in claim 5, the processor saves the generated pixel data frame into the memory as image data. This stored frame can be used later for display or further processing.
7. The system of claim 5 , wherein the processor is further configured to output the frame of pixel data to the projector and the projector is further configured to display the frame in the projected image.
Expanding on the system described in claim 5, the processor sends the generated pixel data frame to the projector, which then displays the frame as part of the projected image. This ensures that the rendered image based on the pen's position is shown in real-time.
8. The system of claim 5 , wherein the frame of pixel data represents a two-dimensional section view of a three-dimensional model represented by the design data.
Referring to the system described in claim 5, the rendered pixel data frame represents a 2D cross-section of a 3D model from the design data. As the user moves the pen, the system updates and projects different cross-sections of the 3D model.
9. The system of claim 1 , wherein the position tracking mechanism is further configured to determine whether the spatially-aware projection pen is within a surface input layer that includes the display surface, wherein user input received via the input mechanism when the spatially-aware projection is within the surface input layer modifies the projected image.
The spatially-aware projection pen system described in claim 1 can detect when the pen is within a designated input layer that includes the display surface. Input received via the activation button when the pen is in this input layer modifies the projected image. This allows the user to interact with and change the projected image.
10. The system of claim 1 , wherein the printed content comprises a second portion of the design data.
In the spatially-aware projection pen system described in claim 1, the printed content on the display surface is a second portion of the CAD design data. Thus the projected image complements and extends already available printed information.
11. The system of claim 1 , further comprising an ink reservoir that is configured to deposit physical ink on the display surface when a tip of the spatially-aware projection pen is in contact with the display surface.
The spatially-aware projection pen system from claim 1 includes a physical ink reservoir that deposits ink onto the display surface when the pen tip is in contact. This provides the user with both physical and virtual ink capabilities.
12. The system of claim 1 , further comprising a network interface that is coupled to the processor and configured to interface with a remote user.
The spatially-aware projection pen system of claim 1 includes a network interface that is connected to the processor and facilitates remote interaction with another user. This enables collaboration and data sharing with remote participants.
13. The system of claim 1 , wherein the projector is positioned within the enclosure to provide a field of view angle of 30 degrees.
In the spatially-aware projection pen system of claim 1, the projector is positioned within the enclosure to provide a field of view angle of 30 degrees. This controls the size and perspective of the projected image.
14. The system of claim 1 , wherein the processor is configured to generate a user interface that is displayed within the projected image by the projector.
In the spatially-aware projection pen system of claim 1, the processor generates a user interface that the projector displays within the projected image. This provides a graphical interface for interacting with the system.
15. The system of claim 1 , wherein the processor is configured to render a sequence of frames of pixel data based on the design data and a path captured by the position tracking mechanism to produce a walk-through animation of a three-dimensional model represented by the design data.
In the spatially-aware projection pen system of claim 1, the processor can create a walkthrough animation of a 3D model. It does this by rendering a sequence of pixel data frames based on the CAD data and the path the user takes with the pen. The projector displays these frames to produce the animation.
16. The system of claim 1 , wherein the processor is configured to generate virtual ink that is displayed in the projected image by the projector when the position tracking mechanism indicates that the spatially-aware projection pen is in contact with the display surface.
In the spatially-aware projection pen system of claim 1, the processor generates virtual ink that the projector displays when the pen's position tracking mechanism detects contact with the display surface. This simulates digital ink strokes in real-time, allowing the user to digitally draw on the surface.
17. A spatially-aware projection pen system, comprising: an input mechanism configured to be activated and deactivated by a user; a camera configured to capture image data and determine a position of the spatially-aware projection pen within a three-dimensional space and relative to a display surface; a memory configured to store design data, image data, and an application program, wherein the design data comprises computer-aided design (CAD) data; a processor configured to: execute the application program to generate at least a portion of the image data based on the CAD data and printed content on the display surface, receive input signals from the input mechanism and the position tracking mechanism, and generate a projected image based on the image data, the input signals and the position of the spatially-aware projection pen, wherein the projected image includes a plurality of visual components, and wherein a geometry and a location of each visual component corresponds to a different content portion of the printed content; a projector configured to display the projected image overlaid on the printed content, wherein a location of the projected image is locked relative to the printed content and remains static as the position of the spatially-aware projection pen changes; and an enclosure configured to be grasped by a human hand and house the input mechanism, the position tracking mechanism, the memory, the projector, and the processor.
A spatially-aware projection pen system includes a pen-shaped device for projecting images onto a surface and augmenting printed content with digital information. The pen features a button for activation, a camera for determining its position in 3D space relative to the display surface, a memory for storing CAD data, image data, and an application. The processor executes the application to generate image data using CAD data and printed content. It processes input from the button and camera to create a projected image containing visual components aligned with the printed content. A projector displays this image overlaid on the printed content, locked to the printed content’s position, regardless of pen movement. All components are enclosed in a hand-held form factor.
18. The system of claim 17 , wherein the camera is further configured to determine the position of the spatially-aware projection pen in a three-dimensional space by retrieving camera calibration parameters based on traceable patterns on the display surface.
Building on claim 17, the camera determines the pen's 3D position using calibration parameters derived from traceable patterns on the display surface. These patterns allow the camera to accurately locate the pen in space.
19. The system of claim 17 , wherein the projector is further configured to generate virtual ink corresponding to pen strokes in the projected image when the camera indicates that the spatially-aware projection pen is in contact with the display surface.
Expanding on claim 17, the projector displays virtual ink corresponding to pen strokes when the camera detects that the pen is touching the display surface. This simulates digital ink in real-time.
20. The system of claim 17 , wherein the camera is further configured to determine whether the spatially-aware projection pen is within a surface input layer that includes the display surface, within a hover input layer of the three-dimensional space that is directly above the display surface, or within a spatial input layer of the three-dimensional space that is directly above the hover input layer.
Building on claim 17, the camera can detect if the pen is within a surface input layer (on the display), a hover input layer (just above the surface), or a spatial input layer (above the hover layer). This allows for different interaction modes depending on the pen's distance from the surface.
21. The system of claim 1 , wherein the position tracking mechanism is further configured to determine whether the spatially-aware projection pen is within a hover input layer that is a fixed distance from the display surface, wherein user input received via the input mechanism when the spatially-aware projection is within the hover input layer manipulates a cursor displayed on the projected image.
Building on claim 1, the system detects when the pen is within a "hover input layer" a fixed distance above the display surface. Activating the pen's button while in this layer manipulates a cursor in the projected image, allowing for interaction without direct surface contact.
22. The system of claim 21 , wherein the position tracking mechanism is further configured to determine whether the spatially-aware projection pen is within a spatial input layer that is a second fixed distance from the hover input layer, wherein second user input received via the input mechanism when the spatially-aware projection is within the hover input layer manipulates a scale of the projected image.
Expanding on the system in claim 21, the system also includes a "spatial input layer" at a second, fixed distance above the hover input layer. Activating the pen's button in the spatial input layer adjusts the scale of the projected image. This allows for zoom and perspective control based on the pen's height.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
August 6, 2009
August 15, 2017
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.