Annotating documents on a mobile device

This application is a continuation of, and claims a benefit of priority under 35 U.S.C. 120 of the filing date of U.S. patent application Ser. No. 17/335,805, filed Jun. 1, 2021, issued as U.S. Pat. No. 11,875,108, entitled “ANNOTATING DOCUMENTS ON A MOBILE DEVICE,” which is a continuation of, and claims a benefit of priority under 35 U.S.C. 120 of the filing date of U.S. patent application Ser. No. 16/835,291, filed Mar. 30, 2020, issued as U.S. Pat. No. 11,030,396, entitled “ANNOTATING DOCUMENTS ON A MOBILE DEVICE,” which is a continuation of, and claims a benefit of priority under 35 U.S.C. 120 of the filing date of U.S. patent application Ser. No. 15/084,000, filed Mar. 29, 2016, issued as U.S. Pat. No. 10,606,941, entitled “ANNOTATING DOCUMENTS ON A MOBILE DEVICE,” which claims the benefit of priority under 35 U.S.C. 119(e) of the filing date of U.S. Provisional Patent Application No. 62/203,359, filed Aug. 10, 2015, entitled “ANNOTATING DOCUMENTS ON A MOBILE DEVICE,” the entire contents of all are hereby expressly incorporated herein by reference for all purposes.

This disclosure relates generally to annotating electronic content, and in particular to integrating content viewing and annotation within a single user interface.

Document collaboration tools are becoming a popular resource for users collaborating on production or editing of electronic content. When collaborating on a document, users often communicate their suggestions or opinions for the document to other users by directly annotating the document or by providing comments via a separate communication channel, such as email. Generally, document collaboration tools enable users to create static annotations of documents. That is, a user can add text, images, or other annotations into the document file that can be distributed to other users, but the other users cannot respond to the annotation without modifying the annotation or adding another annotation to the document. Accordingly, users of the document collaboration tools may use a separate communication channel to discuss the document. The users therefore often switch between applications to view the document and comment on the document, which is inconvenient to the users. In particular, such application switching is challenging on a mobile device with limited display area, which often lack space to simultaneously display both a document and a communication application.

A computing device provides a convenient and easy-to-use interface for creating, modifying, and viewing annotations associated with electronic content. The annotation methods provided by the computing device allow users to manage annotations associated with any of a variety of types of electronic content, such as images, multipage documents, or videos.

The computing device displays the electronic content, which is associated with a plurality of annotations each corresponding to a portion of the content (such as a spatial region, a spatial region at a specified time, or a spatial region on a specified page). The computing device also displays a first annotation, which includes one or more comments received from users and which is associated with a first portion of the electronic content. While displaying the first annotation, an input to display a second annotation is received. The second annotation is associated with a second portion of the electronic content, and includes one or more comments associated with the second portion. In response to receiving the input, the computing device displays the second portion of the electronic content and the one or more comments associated with the second portion.

The features and advantages described in this summary and the following detailed description are not all-inclusive. Many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims.

The figures depict various embodiments of the present disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the disclosure described herein.

illustrates an example system environmentfor annotating documents, according to one embodiment. In one embodiment, the environmentincludes one or more client devicesand a content servercommunicating with the client devicesover a network.

The client devicedisplays electronic content and annotations associated with the content to a user. The client devicemay display and facilitate user annotations of any of a variety of types of electronic content, including documents such as images, slide decks, or PDFs, as well as video or audio files. The client devicemay be any user computing device, such as a desktop computer, a laptop computer, a tablet, or a mobile phone. The client deviceincludes one or more input devices for receiving inputs from a user, such as a touchscreen display or a mouse and keyboard. In one embodiment, the client deviceexecutes an application allowing a user of the client deviceto interact with electronic content. For example, the client deviceexecutes a browser application or a native application that interacts with the content serverto display content stored at the content serverto a user of the client device.

The client deviceenables users to annotate the content and view annotations associated with the content. As described below with respect to, the client deviceintegrates annotation functionality into a content viewer. By providing content viewing and annotation within a single interface, the client deviceenables users to easily annotate electronic content without, for example, switching between applications on the client device. As used herein, an “annotation” is a set of one or more comments from users of the client devicesthat is associated with a specified portion of content. Each comment includes a user response to the specified portion of content, and may include text, images, links, or the like. When creating an annotation, a user of the client devicespecifies the portion of the content with which to associate the annotation. The specified portion may be a spatial area of the content (e.g., a portion of an image), a temporal segment of the content (e.g., a segment of an audio file), a page of the content, both a spatial and temporal portion of the content (e.g., an area within one or more frames of a video), and/or another portion of the content. User comments added to the content are linked to the specified portion. In one embodiment, users can reply to an annotation in the electronic content, creating a comment chain associated with the specified portion of the content and configured for display along with the electronic content.

The client devicecommunicates annotations created by users to the content server, which may display the content and annotations to users of other client devices. Thus, for example, the client deviceenables a user to comment on a specified portion of a document and share the comment with other users via the content server. The client devicesused by the other users display the comment to the other users and indicate the portion of the document with which the comment is associated.

In one embodiment, the types of actions a user may take on a client devicedepend on the configuration of the client device. For example, a mobile device may allow a user to view content and annotations, create new annotations, and add comments to the annotations, but may not allow the user to modify the portion of the content with which an annotation is associated. A larger computing device (e.g., a desktop or laptop computer) may allow the user to also modify the annotation.

The content serverfacilitates sharing of electronic content and associated annotations between the client devices. The content servermay be any content storage device or system, such as a file hosting service, a document management system, or an email server. Users of the client devicesmay share electronic content and associated annotations using the content server. In one embodiment, the content serverstores the electronic content and any annotations associated with the content, and users access the content by directly accessing the content server. For example, a user may share a document with another user by uploading the document to the content serverand sending the other user a link to the document's location at the server. Any users who have the link to the document may access the document via the content serverto view the content of the document and any associated annotations, as well as to create or modify annotations associated with the content. In another embodiment, electronic content and its associated annotations are stored at client devices, and the content serversynchronizes the content by pushing changes made to a content item at a first client device(such as the creation of a new annotation) to one or more other client devicesstoring the content item. In yet another embodiment, the content serverstores annotations associated with electronic content stored at client devices, and synchronizes the annotations with the electronic content at the devices.

The content servermay encrypt the content or otherwise control access to the content by unauthorized users. For example, the content servermay require a user to provide login credentials to access a particular document or set of documents the user has been authorized to access. In one embodiment, users may be provided with different levels of access. For example, some users may be given both read and write access to a content item, permitting the user to view the content item and any associated annotations, edit the content item, and annotate the content item. Other users may be given annotation-only permissions for a content item, allowing the user to add annotations or comments to the content item but not allowing the user to modify the content item itself.

The client devicesand content serverare configured to communicate via the network, which may comprise any combination of local area and/or wide area networks, using both wired and/or wireless communication systems. In one embodiment, the networkuses standard communications technologies and/or protocols. For example, the networkincludes communication links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 3G, 4G, code division multiple access (CDMA), digital subscriber line (DSL), etc. Examples of networking protocols used for communicating via the networkinclude multiprotocol label switching (MPLS), transmission control protocol/Internet protocol (TCP/IP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), and file transfer protocol (FTP). Data exchanged over the networkmay be represented using any suitable format, such as hypertext markup language (HTML) or extensible markup language (XML). In some embodiments, all or some of the communication links of the networkmay be encrypted using any suitable technique or techniques.

illustrate a process performed at the client devicefor creating an annotation associated with electronic content, according to one embodiment. In the example of, the process is described with respect to an imageas an example type of electronic content. However, other content types may be annotated by processes similar to those described herein. As shown in, the client devicedisplays the image. While the imageis displayed, a user inputis received to create an annotation associated with the image. For example,illustrates a touch input (e.g., a press-and-hold input) received at a touchscreen display of the client deviceas the inputto create an annotation, although the input may alternatively be received by other input devices of the client device(such as a mouse).

In one embodiment, the user inputindicates a region of the imagewith which to associate the annotation. In response to the input, the client devicedisplays a text entry boxand a selection box, as shown in. The user can enter text, such as commentary on the image, into the text entry boxas a comment associated with the annotation. The selection boxdefines a region of the imagewith which the user desires to associate the annotation. In one embodiment, the user can change the dimensions of the selection box, increase or decrease a zoom level of the image, and/or move the position of the selection boxor the imageon the display of the client deviceto change the region of the imagethat will be associated with the annotation. Althoughillustrates the selection boxhaving a rectangular shape, the selection boxmay be any of a variety of other shapes such as a circle, a triangle, or a freeform polygon.

illustrates text entered by the user into the text entry box. When the user confirms creation of the annotation (by, for example, selecting the check mark), the client devicecreates an annotation using the text entered by the user and associates the annotation with the portion of the document identified by the selection box. The annotation created by the client deviceis shown in. The client devicesends the text to the content server, which stores the text and a location of the selection boxwithin the imageas an annotation of the image.

illustrate a process performed at the client devicefor adding a comment to an existing annotation associated with electronic content, according to one embodiment.illustrates an annotationassociated with the image. The annotationincludes a comment chain, comprising one or more comments provided by users of the client devices. A user inputto add a comment to the comment chain associated with the annotationis received. For example, the client devicereceives a touch input at an “add a comment” button. In response to the input, the client devicedisplays a text entry box, as shown in, for receiving the comment from the user of the client device.illustrates text entered by the user into the text entry box. When the user confirms the comment (by, for example, selecting the check mark), the client deviceassociates the comment with the annotation.illustrates the comment added to the comment chainassociated with the annotation. Thus, a user of the client devicecan reply directly to a comment associated with the image.

is a flowchart illustrating a processfor displaying annotations associated with electronic content, according to one embodiment. In one embodiment, the processshown inis performed by a client device. The processmay include different, additional, or fewer steps, and the steps may be performed in different orders.is described with respect to the user interfaces shown in.

Referring to, the client devicedisplayselectronic content associated with a plurality of annotations. Each annotation corresponds to a respective portion of the electronic content. Depending on the format of the electronic content, the annotation may correspond to a spatial region within the content, a spatial region within the content at a particular time, or a spatial region on a particular page of the content.illustrates an image, as example electronic content, displayed by the client device. In one embodiment, the client devicealso displays an annotation barindicating a number of annotations associated with the image. The client devicemay indicate the annotations associated with the imagein a variety of other manners, such as displaying numbered or lettered identifiers of the annotations within the image. The annotation barmay also be displayed at a different location on the display than shown in.

The client devicedisplaysa first annotation, which includes one or more comments associated with a first portion of the electronic content. To display an annotation associated with the image, a user may provide an input at the annotation barshown in. For example, the user taps at the annotation baror swipes the annotation barupwards on the display of the client device. As shown in, the client devicein one embodiment displays one or more commentscorresponding to a first annotation in a first regionof the display, and displays a portion of the imagewith which the first annotation is associated in a second regionof the display. The first regionof the display may be scrollable to display additional commentsassociated with the first annotation.

In one embodiment, the client devicedisplays the electronic content at a zoom level such that the first portion is displayed with a border defining a distance between at least one side of the first portion and an edge of the display of the client device. The border may be a fixed size (e.g., 100 pixels), a fixed portion of the display area displaying the first portion (e.g., 10%), or dynamically determined based on the size and shape of the first portion. For example, as shown in, the imagedisplayed in the second regionof the display such that the spatial regionassociated with the first annotation is displayed, as well as a borderspecifying a distance between a side of the spatial regionand an edge of the display. In another embodiment, the client devicedisplays the electronic content at a specified zoom level within the second regionof the display, regardless of the size of the first portion of the electronic content associated with the first annotation.

The client devicereceivesan input to display a second annotation associated with the electronic content. Like the first annotation, the second annotation includes one or more comments and is associated with a second portion of the electronic content.illustrates an example user inputreceivedby the client deviceto navigate to a second annotation associated with the image. In the example of, the user inputis a swipe gesture received in the first regionof the display. In other embodiments, a user may navigate to a second annotation by repositioning the image within the second regionof the display to the location associated with the second annotation, selecting the second annotation from a menu, selecting an arrow key of the client deviceor an arrow button displayed by the client device, or providing any of a variety of other types of input.

In response to the user input, the client devicedisplaysthe second portion of the electronic content and the one or more comments associated with the second annotation.illustrates example commentsassociated with a second annotation that are displayed in the first regionof the display, and a portion of the imagewith which the second annotation is associated displayed in the second regionof the display. The portion of the image displayed in the second regionof the display includes the second portionwith which the second annotation is associated. In one embodiment, the client deviceautomatically pans and/or zooms the electronic content to display the second portion of the content associated with the second annotation, without a manual input from the user to pan or zoom the image. When transitioning to the second annotation, the client devicemay pan or zoom the electronic content until the second portion of the content is displayed at a predetermined position within the first regionof the display. For example, as shown in, the second portionis displayed with the borderdefining a distance between a right side of the second portionand a right edge of the display. The client devicemay additionally or alternatively display the second portionwith borders on other sides, or may display the electronic content at a predetermined zoom level regardless of the size of the second portion.

illustrate an example of the processwhen the electronic content is a multipage document and annotations are associated with different pages of the document.

As shown in, the client devicedisplays a first annotation associated with a first portionof the document. In the example of, the first portionis a portion of pageof the document. The first annotation includes at least one comment, which in one embodiment is displayed in the first regionof the display while the portionof the document is displayed in the second region.

When transitioning from the first annotation to a second annotation on a different page (e.g., in response to a user swipe gesture received at the first regionof the display), the client deviceshows an animation including content of the document between the portion associated with the first annotation and the portion associated with the second annotation. In one embodiment, as shown for example in, the animation zooms out from the first portion of the document until one or more pages are displayed, scrolls past representations of pages of the document between the first and second annotations, and zooms into the portion of the document associated with the second annotation.

illustrates an example of the client devicezooming out from the portionof the document associated with the first annotation, andillustrates the display zoomed out to show a representationof the entire first page of the document. Althoughshows an example in which the client devicezooms out the displayed portion of the document until one document page is shown in its entirety, the zoom level of document pages during the transition to the second annotation may be greater or lesser than shown in. For example, the client devicemay zoom out until multiple pages are displayed.

illustrates the client devicescrolling past representationsof pages of the document between the first page and page, where the second annotation is located. In one embodiment, the client devicedisplays the intervening pages if the pages are cached at the client deviceor retrieved quickly enough from the content serverto display during the annotation. If the pages are not locally cached and the network connection between the client deviceand the serveris slower, the client devicemay instead display blank page outlines of the intervening pages or other representations of the pages.

illustrates the client devicebeginning to zoom in on a second portionof the document with which the second annotation is associated, andillustrates the display zoomed in on the second portion. In one embodiment, the client devicezooms in on the portionuntil it is displayed at a specified zoom level. In another embodiment, the client devicezooms in until the portionis displayed with a border defining a distance between at least one side of the portionand an edge of the display. While the portionof the document is displayed (e.g., in the second regionof the display), the client devicealso displays the one or more commentsassociated with the second annotation.

The processes described with respect toand illustrated inenable a user to easily view and create annotations associated with electronic content. Because a communication channel is integrated into a content viewer, the user of the client devicedoes not need to switch between applications to view the content and add comments to the content. This integration of a communication channel into a content viewer is particularly advantageous when the client deviceis a mobile device with a relatively small display area (e.g., a mobile phone), since switching between applications on a mobile device significantly increases the cognitive burden to the user and the amount of time the user would need to spend to annotate the content. Moreover, by linking an annotation to a specified portion of the content and showing the specified portion while displaying the annotation, the client deviceenables a user to create annotations that specifically identify the portion of the content of interest to the user. These annotations enable the user to more clearly communicate comments on the electronic content with other users, improving user workflow when collaborating on documents.

The foregoing description of the embodiments of the disclosure has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of this description describe the embodiments of the disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a tangible computer readable storage medium or any type of media suitable for storing electronic instructions, and coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Embodiments of the disclosure may also relate to a computer data signal embodied in a carrier wave, where the computer data signal includes any embodiment of a computer program product or other data combination described herein. The computer data signal is a product that is presented in a tangible medium or carrier wave and modulated or otherwise encoded in the carrier wave, which is tangible, and transmitted according to any suitable transmission method.

Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the disclosure be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments of the disclosure is intended to be illustrative, but not limiting, of the scope of the invention.