Synchronizing an external location

This application is a continuation and claims the benefit of U.S. application Ser. No. 17/855,842, filed on Jul. 1, 2022, entitled “SYNCHRONIZING AN EXTERNAL LOCATION,” which is a continuation and claims the benefit of U.S. application Ser. No. 17/132,954, filed on Dec. 23, 2020, entitled “SYNCHRONIZING AN EXTERNAL LOCATION,” which claims priority to U.S. Provisional Patent Application No. 63/093,699, filed on Oct. 19, 2020, entitled “SYNCHRONIZING AN EXTERNAL LOCATION.” All of which are expressly incorporated by reference herein in their entireties.

Content management systems allow users to store and manage all necessary content items in one place. Content management systems generally store content items on servers and allow users access to the content items over a network. Some content management systems allow users to synchronize content items stored on the local client device to the content management system. However, content management systems do not offer synchronization of content items stored on an external drive attached to a client device.

The present technology is capable of synchronizing objects that are located outside of a synchronized directory of a content management system. Content management systems that provide for synchronization between cloud storage and a client device synchronize contents into a particular directory on the client device. Objects located outside of a synchronized directory are typically not accessible to a synchronization service of a content management system. However, some of a user's most important objects might be located outside of the synchronized directory and therefore there is a need to be able to synchronize such content items.

Objects stored in an external directory on an external hard drive present an even bigger challenge, and yet objects on an external hard drive might be some of a user's most important to backup through a synchronized content management system. For example, users commonly utilize external hard drives to archive important objects that are not needed on a daily basis, but are none the less important. External hard drives provide complications in that such hard drives might not always be accessible to a computer running a synchronization client application, they might be formatted differently than the hard drive of the client device, and they might be read-only.

A further problem can arise when an external hard drive stores a greater amount of bytes than the available storage space on a client computer's hard drive. In such instances, objects cannot be first stored on the hard drive of the client prior to being uploaded to a synchronized content management system.

The present technology overcomes many of these problems and allows for an external directory, even on external hard drives, to be synchronized to a content management system without first copying the objects to be synchronized to a location on a client device's hard drive.

In some embodiments, the disclosed technology is deployed in the context of a content management system having object synchronization capabilities and collaboration features, among others. An example system configurationis shown in, which depicts content management systeminteracting with a client device.

Accounts

Content management systemcan store objects in association with accounts, as well as perform a variety of object management tasks, such as retrieve, modify, browse, and/or share the object(s). Furthermore, content management systemcan enable an account to access object(s) from multiple client devices.

Content management systemsupports a plurality of accounts. A subject (user, group, team, company, etc.) can create an account with content management system, and account details can be stored in subject database. Subject databasecan identify a registered subject by a subject ID, and store profile information for registered subjects in association with the subject ID. In some cases, profile information for registered subjects includes a subject name and/or email address. Subject databasecan include account management information, such as account type (e.g., various tiers of free or paid accounts), storage space allocated, storage space used, client deviceshaving a registered content management client applicationresident thereon, security settings, personal configuration settings, etc. In some embodiments, some information associated with an account may not be directly stored, and rather this information can be derived. For example, storage space used might be explicitly stored, or it can be calculated when needed.

In some embodiments, subject databaseneed not store complete information associated with an account for a subject. Some portion of information associated with an account for a subject can also be stored in another database of content management systemsuch as metadata database, or in a database external to content management system.

Subject databasecan store groups of accounts associated with a subject. Groups can have permissions based on group permissions statements and/or access control lists, and members of the groups can inherit the permissions. For example, a marketing group can have access to one set of objects while an engineering group can have access to another set of objects. An administrator group can modify groups, modify subject accounts, etc. Groups are also subjects identified by subject ID.

In some embodiments, subject databasecan be broken into a plurality of tables, indexes and other data structures.

Object Storage

A feature of content management systemis the storage of objects, which can be stored in object storage. An object generally is any entity that can be recorded in a file system. Objects can be any object including digital data such as documents, collaboration objects, text files, audio files, image files, video files, webpages, executable files, binary files, directories, folders, zip files, playlists, albums, etc.

In some embodiments, objects can be grouped into a collection, which can refer to a folder including a plurality of objects, or a plurality of objects that are related or grouped by a common attribute.

In some embodiments, object storageis combined with other types of storage or databases to handle specific functions. Object storagecan store objects, while metadata regarding the objects can be stored in metadata database. Likewise, data regarding where an object is stored in object storagecan be stored in content directory. Additionally, data regarding changes, access, etc. can be stored in object database. Object databasecan also include a subject account identifier that identifies the subject IDs that have access to the object.

In some embodiments, object databasecan be broken into a plurality of tables, indexes and other data structures.

Each of the various storages/databases such as object storage, content directory, object database, and metadata databasecan be comprised of more than one such storage or database and can be distributed over many devices and locations. Other configurations are also possible. For example, data from object storage, content directory, object database, and/or metadata databasemay be combined into one or more object storages or databases or further segmented into additional object storages or databases. Thus, content management systemmay include more or less storages and/or databases than shown in.

In some embodiments, object storageis associated with at least one object storage service, which includes software or other processor executable instructions for managing the storage of objects including, but not limited to, receiving objects for storage, preparing objects for storage, selecting a storage location for the object, retrieving objects from storage, etc. In some embodiments, object storage servicecan divide an object into smaller chunks for storage at object storage. The location of each chunk making up an object can be recorded in content directory. Content directorycan include a content entry for each object stored in object storage. The content entry can be associated with an object ID, which identifies an object.

In some embodiments, each object and each chunk of an object can also be identified from a deterministic hash function. This method of identifying an object and chunks of objects can ensure that object duplicates are recognized as such since the deterministic hash function will output the same hash for every copy of the same object, but will output a different hash for a different object. Using this methodology, object storage servicecan output a unique hash for each different version of an object.

Object storage servicecan also designate or record a content path for an object in object database. The content path can include the name of the object and/or folder hierarchy associated with the object. For example, the content path can include a folder or path of folders in which the object is stored in a local file system on a client device. In some embodiments, object database might only store a direct ancestor or direct child of any object, which allows a full path for an object to be derived.

While objects are stored in object storagein blocks and may not be stored under a tree like directory structure, such directory structure is a comfortable navigation structure for subjects. Object storage servicecan define or record a content path for an object wherein the “root” node of a directory structure can be any directory with specific access privileges assigned to it, as opposed to a directory that inherits access privileges from another directory.

In some embodiments, a root directory can be mounted underneath another root directory to give the appearance of a single directory structure. This can occur when an account has access to a plurality of root directories. As addressed above, the directory structure is merely a comfortable navigation structure for subjects, but does not correlate to storage locations of objects in object storage.

While the directory structure in which an account views objects does not correlate to storage locations at content management system, the directory structure can correlate to storage locations on client devicedepending on the file system used by client device.

As addressed above, a content entry in content directorycan also include the location of each chunk making up an object. More specifically, the content entry can include content pointers that identify the location in object storageof the chunks that make up the object.

Object storage servicecan decrease the amount of storage space required by identifying duplicate objects or duplicate blocks that make up an object or versions of an object. Instead of storing multiple copies, object storagecan store a single copy of the object or block of the object and content directorycan include a pointer or other mechanism to link the duplicates to the single copy.

Object storage servicecan also store metadata describing objects, object types, folders, file paths, and/or the relationship of objects to various accounts, collections, or groups in metadata database, in association with the object ID of the object.

Object storage servicecan also store a log of data regarding changes, access, etc. in object database. Object databasecan include the object ID of the object and can optionally include a description of the change or access action along with a time stamp or version number and any other relevant data. Object databasecan also include pointers to blocks affected by the change or object access. Object storage service can provide the ability to undo operations, by using an object version control that tracks changes to objects, different versions of objects (including diverging version trees), and a change history that can be acquired from object database.

Object Synchronization

Another feature of content management systemis the synchronization of objects with at least one client device. Client device(s) can take different forms and have different capabilities. For example, client device-is a computing device having a local file system accessible by multiple applications resident thereon. Client device-is a computing device wherein objects are only accessible to a specific application or by permission given by the specific application, and the objects are typically stored either in an application specific space or in the cloud. Client device-is any client device accessing content management systemvia a web browser and accessing objects via a web interface. While example client devices-,-, and-are depicted in form factors such as a laptop, mobile device, or web browser, it should be understood that the descriptions thereof are not limited to devices of these example form factors. For example, a mobile device such as client-might have a local file system accessible by multiple applications resident thereon, or client-might access content management systemvia a web browser. As such, the form factor should not be considered limiting when considering client's capabilities. One or more functions described herein with respect to client devicemay or may not be available on every client device depending on the specific capabilities of the device—the file access model being one such capability.

In some embodiments, client devicecan be communicatively connected to an external hard drive.

In many embodiments, client devices are associated with an account of content management system, but in some embodiments, client devices can access content using shared links and do not require an account.

As noted above, some client devices can access content management systemusing a web browser. However, client devices can also access content management systemusing client applicationstored and running on client device. Client applicationcan include a client synchronization service.

Client synchronization servicecan be in communication with server synchronization serviceto synchronize changes to objects between client deviceand content management system.

Client devicecan synchronize content with content management systemvia client synchronization service. The synchronization can be platform agnostic. That is, content can be synchronized across multiple client devices of varying types, capabilities, operating systems, etc. Client synchronization servicecan synchronize any changes (new, deleted, modified, copied, or moved objects) to objects in a designated directory of a file system of client devicethat is monitored by synchronization service.

Objects can be synchronized from client deviceto content management system, and vice versa. In embodiments wherein synchronization is from client deviceto content management system, a subject can manipulate objects directly from the file system of client device, while client synchronization servicecan monitor a designated directory on client devicefor changes to files within the monitored folders.

When client synchronization servicedetects a write, move, copy, or delete of content in a directory that it monitors, client synchronization servicecan synchronize the changes to object storage service. In some embodiments, client synchronization servicecan perform some functions of object storage serviceincluding functions addressed above such as dividing the object into blocks, hashing the object to generate a unique identifier, etc. Client synchronization servicecan index content within client storage indexand save the result in storage index. Indexing can include storing paths plus the object identifier, and a unique client identifier for each object. In some embodiments, client synchronization servicelearns the object identifier from server synchronization service, and learns the unique client identifier from the operating system of client device.

Client synchronization servicecan use storage indexto facilitate the synchronization of at least a portion of the objects within client storage with objects associated with a subject account on content management system. For example, client synchronization servicecan compare storage indexwith content management systemand detect differences between content on client storage and content associated with a subject account on content management system. Client synchronization servicecan then attempt to reconcile differences by uploading, downloading, modifying, and deleting content on client storage as appropriate. Object storage servicecan store the changed or new block for the object and update object database, metadata database, content directory, object storage, subject database, etc. as appropriate.

When synchronizing from content management systemto client device, data regarding a mount, modification, addition, deletion, move of an object recorded in object databasecan trigger a notification to be sent to client deviceusing notification service. When client deviceis informed of the change, client devicecan make a request for changes listed in object databasesince the last synchronization point known to the client device. When client devicedetermines that it is out of synchronization with content management system, client synchronization servicerequests object blocks including the changes, and updates its local copy of the changed objects.

In some embodiments, storage indexstores tree data structures wherein one tree reflects the latest representation of a directory according to server synchronization service, while another tree reflects the latest representation of the directory according to client synchronization service. Client synchronization service can work to ensure that the tree structures match by requesting data from server synchronization serviceor committing changes on client deviceto content management system.

Sometimes client devicemight not have a network connection available. In this scenario, client synchronization servicecan monitor the linked collection for object changes and queue those changes for later synchronization to content management systemwhen a network connection is available. Similarly, a subject can manually start, stop, pause, or resume synchronization with content management system.

Client synchronization servicecan synchronize all content associated with a particular subject account on content management system. Alternatively, client synchronization servicecan selectively synchronize some of the objects associated with the particular subject account on content management system. Selectively synchronizing only some of the objects can preserve space on client deviceand save bandwidth.

In some embodiments, client synchronization serviceselectively stores a portion of the objects associated with the particular subject account and stores placeholder objects in client storage for the remaining portion of the objects. For example, client synchronization servicecan store a placeholder object that has the same filename, path, extension, metadata, of its respective complete object on content management system, but lacking the data of the complete object. The placeholder object can be a few bytes or less in size while the respective complete object might be significantly larger. After client deviceattempts to access the object, client synchronization servicecan retrieve the data of the object from content management systemand provide the complete object to client device. This approach can provide significant space and bandwidth savings while still providing full access to a subject's objects on content management system.

Collaboration Features

Another feature of content management systemis to facilitate collaboration between subjects. Collaboration features include object sharing, commenting on objects, co-working on objects, instant messaging, providing presence, and seen state information regarding objects, etc.

Sharing

Content management systemcan manage sharing objects via sharing service. Sharing an object by providing a link to the object can include making the object accessible from any computing device in network communication with content management system. However, in some embodiments a link can be associated with access restrictions enforced by content management systemand Identity and Access Management (IAM) service. Sharing content can also include linking content using sharing serviceto share content within content management systemwith at least one additional subject account (in addition to the original subject account associated with the object) so that each subject account has access to the object. The additional subject account can gain access to the content by accepting the content, which will then be accessible through either web interface serviceor directly from within the directory structure associated with their account on client device. The sharing can be performed in a platform-agnostic manner. That is, the content can be shared across multiple client devicesof varying types, capabilities, operating systems, etc. The content can also be shared across varying types of subject accounts.

To share an object within content management systemsharing servicecan add a subject ID of a team or of one or more subject accounts with a content item in objects databaseassociated with the object, thus granting the added subject account(s) access to the object. Sharing servicecan also remove subject IDs from being permitted to access a content item in objects databaseto restrict a subject account's access to the object. Sharing servicecan record object identifiers, subject identifiers given access to an object, and access levels in objects database. For example, in some embodiments, subject identifiers associated with a single content item can specify different permissions for respective subject identifiers with respect to the associated object.

In some embodiments, content management systemcan include an access control listwhich includes a description of complete access rights pertaining to a respective object. An access control list for any respective object in content management system can be derived from objects database. In some embodiments, it is not desirable to maintain a persistent access control listfor a respective object, as an access control listfor a respective object can be derived when needed.