Executable objects in a distributed storage system

The present description relates to running executable objects, and more specifically, to a system, method, and machine-readable storage medium for running executable objects in a distributed storage system for cost savings and/or improved efficiency.

Networks and distributed storage allow data and storage space to be shared between devices located anywhere a connection is available. These implementations may range from a single machine offering a shared drive over a home network to an enterprise-class cloud storage array with multiple copies of data distributed throughout the world. Larger implementations may incorporate Network Attached Storage (NAS) devices, Storage Area Network (SAN) devices, and other configurations of storage elements and controllers to provide data and manage its flow. Improvements in distributed storage have given rise to their use across many industries and applications that make regular use of the stored data.

Some cloud providers may charge for the amount of storage used which may include transferring the data to the storage. Some cloud providers may also charge to transfer data out of the storage. In current approaches, deriving value from the stored data typically involves transferring the data from storage to a compute node to be processed or analyzed. As the size and amount of data stored increases, so too does the cost of deriving value from that data, both financially as well as in terms of resource use. For example, transferring data utilizes compute resources, memory resources, and network resources, and can be both time-consuming and resource-intensive because of the extra copying and processing (including memory copies, network transfers, temporary disk copies, etc.).

Moreover, for many object storage applications the object storage node(s) have unused compute and memory resources, such as after storing a large amount of data in the storage node(s). These unused resources represent a lost opportunity for better resource utilization, as well as cost and resource efficiencies by reducing the amount of data transferred out of, and into, the storage.

The following summarizes some aspects of the present disclosure to provide a basic understanding of the discussed technology. This summary is not an extensive overview of all contemplated features of the disclosure and is intended neither to identify key or critical elements of all aspects of the disclosure nor to delineate the scope of any or all aspects of the disclosure. Its sole purpose is to present some concepts of one or more aspects of the disclosure in summary form as a prelude to the more detailed description that is presented later.

For example, in an aspect of the disclosure, a method includes storing, by a node in an object storage system, an executable object in the node. The method further includes receiving, by the node, a request to run the executable object. The method further includes identifying, by the node, one or more data objects stored on the node to be processed by the executable object. The method further includes aggregating, by the node, the one or more data objects for processing by the executable object. The method further includes running, by the node, the executable object on the node to process the one or more data object stored on the node.

In an additional aspect of the disclosure, a computing device includes a memory containing having stored thereon instructions for performing a method of running an executable object on an object storage system. The computing device further includes a processor coupled to the memory, the processor configured to receive, by a first node of the object storage system, an execute instruction for running an executable object to process one or more data objects of interest. The processor is further configured to identify, by the first node, a first set of the one or more data objects to be processed by the executable object, the first set of the one or more data objects being stored on the first node with the executable object. The processor is further configured to receive, by the first node, a request from a second node of the object storage system for the executable object in response to a second set of the one or more data objects being located at the second node. The processor is further configured to send, by the first node, a copy of the executable object to the second node in response to the request from the second node. The processor is further configured to run, by the first node, the executable object to process the first set of the one or more data objects stored on the first node in response to being in an active execution state.

In an additional aspect of the disclosure, a non-transitory machine-readable medium having stored thereon instructions for performing a method of running an executable object on an object storage system, when executed by at least one machine, causes the at least one machine to receive, by a first node of the object storage system, execute instructions from a client for running an executable object on the first node, the execute instructions being received via the object storage system. The instructions, when executed by the at least one machine, further causes the at least one machine to identify, by the first node, a first set of data objects to be processed by the executable object. The instructions, when executed by the at least one machine, further causes the at least one machine to run, by the first node, the executable object to process the first set of data objects that are on the first node with the executable object. The instructions further cause the at least one machine to receive, by the first node, a request from the client for information about the first set of data objects while running the executable object, the request the client bypassing the object storage system. The instructions further cause the at least one machine to respond, by the first node, to the request from the client, the response bypassing the object storage system when sent to the client.

Other aspects will become apparent to those of ordinary skill in the art upon reviewing the following description of exemplary embodiments in conjunction with the figures. While one or more embodiments may be discussed as having certain advantageous features, one or more of such features may also be used in accordance with the various embodiments of the invention discussed herein. In similar fashion, while exemplary embodiments may be discussed below as device, system, or method embodiments, it should be understood that such exemplary embodiments can be implemented in various devices, systems, and methods.

All examples and illustrative references are non-limiting and should not be used to limit the claims to specific implementations and embodiments described herein and their equivalents. For simplicity, reference numbers may be repeated between various examples. This repetition is for clarity only and does not dictate a relationship between the respective embodiments. Finally, in view of this disclosure, particular features described in relation to one aspect or embodiment may be applied to other disclosed aspects or embodiments of the disclosure, even though not specifically shown in the drawings or described in the text.

Various embodiments include systems, methods, and machine-readable media for running executable objects on storage nodes of an object storage system. These executable objects, also identifiable as active objects, is a paradigm and framework that enables the use of unused compute and RAM capacity on object storage nodes to efficiently derive value from storage data, increasing the value of the object storage solution to users and avoiding resource consumption that would otherwise have been imposed by data access protocols if that same data processing were performed by an entity external to the object storage system.

In detail, these and other aspects may be accomplished by storing executable code as an object in the object storage system, identifying the object as an executable object, identifying data objects of interest to be processed by the executable object, and running the executable object on the object storage system. By running the executable object at the storage node of the object storage system where the data objects of interest are kept, overall performance of the system may be improved by not transferring data objects out of the data object storage system to be processed. In general, while embodiments of the present disclosure may be discussed with reference to storage nodes of an object storage system, this is for simplicity of discussion only. Embodiments of the present disclosure are applicable to object storage systems generally, such as cloud storage platforms, server storage platforms, on-premises storage platforms, etc. The examples described below may refer to storage nodes to provide illustration of details to the disclosure.

Object storage systems differ from file storage systems in structure and operation. A file storage system stores data files and executables in a directory based hierarchical filesystem. A hierarchical filesystem is a tree structure including directories, data files, and executables that are nested within another directory. Each directory, data file, or executable is addressable using the hierarchical structure of the filesystem. This allows for data files and executables to have the same name as other data files and executables if stored in different directories. Filesystems allow for data files and executables to be coresident in storage (e.g., hard drive (HDD), solid state drive (SSD), CD-ROM, etc.). The executables are run on a central processing unit (CPU) and may process and/or manipulate data files stored within the filesystem. For example, the CPU of a personal computer may copy the executable from storage to random access memory (RAM) or cache to be run. The CPU, running the executable, may modify and/or process the data files in storage.

An object storage system stores data as objects in a flat address space. Objects stored in a flat address space are all stored at the same level, therefore this no nesting or directories. Each object is directly accessible without the need for a hierarchical structure; hence each object name must be unique form all other object names. Objects may be stored in buckets which are logical containers for storing objects. Buckets may be associated with policies that determine which actions a user can perform on a bucket and the objects contained within the bucket. Further, each object includes the object itself and the metadata about the object. As will be discussed in further detail below, object storage systems may include multiple nodes located in physically different locations. As an analogy, an object storage system may be considered similar to an HDD in that it is used for storing data. When data is to be processed, the data must be copied from the object storage system to another system that is typically referred to as a compute node. The compute node may process and/or modify the data that is copied from the object storage system.

This process of copying data housed in an object storage system to a compute node can be time-consuming, resource-intensive, and financially expensive. These problems may be considered part of doing business in order to derive value from the data stored in object storage. As described further below, object storage systems include resources, such compute and memory resources, for storing and retrieving data from the object storage system. These resources may remain idle when no data is being stored or retrieved. Storing executable code in executable objects on the object storage system to be run using the otherwise idle resources of the object storage system may reduce the amount of time, resources, and cost required to derive value from data stored in the object storage system.

Objects storing executable code will be referred to as executable objects going forward to distinguish from objects that contain data, which may be referred to as data objects or as objects. Metadata of the executable objects may be used to mark them as executable. The executable code of the executable object may be configured to execute within the nodes of an object storage system (e.g., storage nodes, cloud storage clusters, etc.) so that processing and changes to data objects occurs within the storage node. In some examples, the executable objects may use protocols for accessing the data objects that are different than the object storage protocols through which the data objects are typically accessed.

In further examples, existing object storage protocols may be modified (e.g., extended) to allow for the creation of executable objects, as well as for the execution (e.g., running) of executable objects within the object storage system (i.e., instead of transferring to a compute node for processing). Some examples of object storage protocols include S3, Azure Blob, and Swift. Such modifications, or extensions, to existing protocols may allow a client to identify an object as executable and control the execution of the executable object against a set of defined data objects. The protocol extensions may further allow a client to control permissions of the executable objects as well as their execution. While such extensions are not necessary for running executable objects, they may reduce the chance of collisions between existing object storage protocol use and the use of executable objects that may occur by overloading existing protocol(s). Such collisions in use may result in errors occurring within the object storage system, which may be avoided by these extensions.

The object storage protocol may include data selection criteria used to identify data objects of interest within the storage system (e.g., within storage nodes, cloud storage clusters, etc.) to be processed by the executable object. Aspects of the present disclosure provide for the generation of data location maps for data objects that are stored in the object storage system (e.g., on a per-node basis). For example, each node within the object storage system may maintain an index of objects stored within the node. The data location maps may be generated by filtering the object index to identify data objects within the node that match the specified data selection criteria. Examples of data selection criteria may include bucket names, object names, prefixes, tags, and/or other metadata. In some examples, the object storage system may further aggregate the data maps of the respective nodes of the system to provide a single cluster wide view of the data location maps. The executable object may use the data location maps during operation to find the data objects of interest.

In further examples, the object storage system may generate an overlay filesystem view from the physical location data maps to be used by the executable objects. In some examples, the overlay filesystem view may be generated from previously generated data maps. The generated overlay filesystem view may allow for efficient reuse of older, legacy code within the executable objects by converting the flat address space of the object storage system to a synthetic hierarchical filesystem view that older code was designed to use. For example, the synthetic filesystem view may use characters in the object name, bucket name, and/or metadata as delimiters, thereby creating synthetic filesystem directories. The synthetic hierarchical filesystem view enables the executable object to access the data object in place (e.g., “zero copy” access) through the use of kernel level translations from the synthetic filesystem view to the object data as it exists on the disk. This overcomes one of the largest limitations of object storage protocols by eliminating resources spent sending and receiving object data using standard protocols. Furthermore, this provides the advantage of presenting all data objects to a user as existing in a single volume while being physically located across different nodes of the object storage system.

In addition to the aspects described above, or alternatively, executable objects according to some embodiments of the present disclosure may be replicated from one node of the object storage system to another node of the object storage system in order to facilitate processing of the data objects at the nodes in which they are physically located. In some examples, the object storage system may copy the executable object to all relevant nodes of the object storage system. In some other examples, nodes that have identified at least one data object to be processed may request, or pull, a copy of the executable object from another node which has a copy of the executable object already.

An executable object may further be associated with a client-side application. For example, an application running outside of the object storage system may be designed to work with an executable object running inside the object storage system to process data stored in the object storage system. The executable objects may be used to maintain a partially processed, or application aware, index of data stored in the object storage system, including the location of the data. The executable objects may communicate directly with the external application without using object storage protocol(s) (e.g., S3, Azure, Swift, etc.). Direct communication between the client-side application and the executable object may allow for pre-loading of frequently used objects and improved latency and performance as compared to traditional object retrieval mechanisms (e.g., object storage protocols).

One or more of the aspects described above may be applied in a manner that improves upon existing pipelining paradigms. For example, executable objects may be used to provide in-place data-centric pipelining. That is, instead of moving data out of the object storage system to compute nodes for processing by executables (as with existing pipelining paradigms), executable objects may be moved into the object storage system to process the data in place according to the embodiments introduced above and further described in various examples below. For example, similar to a data object, a client (e.g., host, client-side application, user, etc.) may use an object storage protocol (e.g., S3, Azure, Swift, etc.) to send a request to store an executable object to the object storage system. Because the size of the executable objects will generally be much smaller than the size of the data being processed, efficiency improvements are realized by not copying data out of the object storage system. The efficiency improvements include reduced network use and latency improvements. In some examples, the executable objects may be spawned on multiple nodes to process the data objects at those nodes in parallel to each other. In some other examples, the executable objects may be spawned on multiple nodes (e.g., in parallel or serially) and process the data serially to each other. This may include processing a first portion of data on a first node (e.g., by running the executable object at the first node until processing of that first portion of data is completed), transferring an execution state of the executable object to a second node where a second portion of data resides, and then processing data on the second node.

Object storage systems according to embodiments of the present disclosure provide improved processing capabilities to users over current object storage systems. Executable objects improve the speed and efficiency by which data may be processed in an object storage system as compared to systems without executable objects (since systems without executable objects process data by transferring the data from storage to a compute node to be processed or analyzed, and then returned, etc.). The data may be processed by the object storage system without expending the time and resources necessary to copy data from the object storage system to a compute node. The time and resource savings increase as the amount of data to be processed increases. Moreover, the use of executable objects may provide the end user with significant cost savings for object storage systems whose cost structures are at least partially based on the amount of data transferred out of the object storage system.

1 FIG. 100 100 102 104 104 104 104 106 106 108 106 108 108 106 106 106 106 106 102 is a schematic diagram of a computing architectureaccording to one or more aspects of the present disclosure. The computing architectureincludes one or more host systems(hosts), each of which may interface with a distributed storage systemto store and manipulate data. The distributed storage systemmay use any suitable architecture and protocol. For example, in some embodiments, the distributed storage systemis a StorageGRID® system, an OpenStack® Swift system, a Ceph system, or other suitable system. The distributed storage systemincludes one or more storage nodesover which the data is distributed. The storage nodesare coupled via a back-end network, which may include any number of wired and/or wireless networks such as a Local Area Network (LAN), an Ethernet subnet, a PCI or PCIe subnet, a switched PCIe subnet, a Wide Area Network (WAN), a Metropolitan Area Network (MAN), the Internet, or the like. In some embodiments, the storage nodesare coupled by a transmission control protocol/Internet protocol (TCP/IP) back-end network, which may be local to a rack or datacenter, although additionally or in the alternative, the networkmay extend between sites in a WAN configuration or be a virtual network extending throughout a cloud. As can be seen, the storage nodesmay be as physically close or as widely dispersed as the application may warrant. In some examples, the storage nodesare housed in the same racks. In other examples, storage nodesare located in different facilities at different sites anywhere in the world. The node arrangement may be determined based on cost, fault tolerance, network infrastructure, geography of the hosts, and/or other considerations. As will be discussed throughout, the methods and systems disclosed herein improve the ability to derive value from the data stored in the storage nodesregardless of the physical distance between the different storage nodes(as well as from hosts).

100 106 102 106 102 100 102 104 106 104 102 102 104 110 106 102 102 104 102 110 104 102 106 102 In the illustrated embodiment, the computing architectureincludes a plurality of storage nodesin communication with a plurality of hosts. It is understood that for clarity and ease of explanation, only a limited number of storage nodesand hostsare illustrated, although the computing architecturemay include any number of hostsin communication with a distributed storage systemcontaining any number of storage nodes. An example storage systemreceives data transactions, executable object manipulation instructions, and/or execute instructions from the hosts. If a data transaction (e.g., requests to read and/or write data) is received from the hosts, the storage systemtakes an action such as reading, writing, or otherwise accessing the requested data so that the storage devicesof the storage nodesappear to be directly connected (local) to the hosts. This allows an application running on a hostto issue transactions directed to the data of the distributed storage systemand thereby access this data as easily as it can access data on storage devices local to the host. In that regard, the storage devicesof the distributed storage systemand the hostsmay include hard disk drives (HDDs), solid state drives (SSDs), storage class memory (SCM), RAM drives, optical drives, and/or any other suitable volatile or non-volatile data storage medium. Further, one or more of the storage nodesmay be connected to one or more cloud storage providers according to embodiments of the present disclosure, and likewise appear to be directly connected (local) to the hosts.

104 106 102 102 104 106 104 102 102 106 106 110 106 106 If an executable object manipulation instruction is received (e.g., to store an executable object at one or more storage nodes, replicate an executable object at other storage nodes, generate or update an overlay filesystem view at one or more nodes, etc.), the storage system(e.g., one or more storage nodes) may perform the requested operation(s). This allows a hostto manipulate and control where and how executable objects run, according to the present disclosure. If an execute instruction (e.g., run executable code of an executable object to process stored data) is received from the hosts, the storage system(e.g., one or more storage nodes) may execute the executable code of the executable object so that the storage systemappears to be one or more additional compute nodes that are connected to the hosts. This allows an application that is running on a hostto request that a storage noderun the executable code of the executable object to process data stored on the storage node(e.g., stored on the storage devices). Further, one or more storage nodesmay receive instructions to run different instances of the same executable code (e.g., copies of an executable object) to process data stored on the one or more storage nodes.

106 106 110 112 112 106 112 112 110 102 110 102 106 106 110 With respect to the storage nodes, an exemplary storage nodecontains any number of storage devicesin communication with one or more storage controllers. The storage controllersmay, according to aspects of the present disclosure, handle the data transactions and execute instructions directed to the storage nodeon which the storage controllersreside. The storage controllersmay also handle data transactions by exercising low-level control over the storage devicesin order to execute (perform) data transactions on behalf of the hosts, and in so doing, may group the storage devices for speed and/or redundancy using a protocol such as RAID (Redundant Array of Independent/Inexpensive Disks). The grouping protocol may also provide virtualization of the grouped storage devices. At a high level, virtualization includes mapping physical addresses of the storage devices into a virtual address space and presenting the virtual address space to the hosts, other storage nodes, and other requestors. In this way, the storage noderepresents the group of storage devices as a single device, often referred to as a volume. Thus, a requestor can access data within a volume without concern for how it is distributed among the underlying storage devices.

112 110 106 102 112 110 110 106 106 106 106 Further, the storage controllersmay handle execute instructions by executing (e.g., running) executable code of one or more executable objects stored on the storage devicesof the storage nodeon behalf of the hosts. The storage controllersmay read the executable code from the executable object from the storage devicesand execute the code. The executable code may include instructions for processing data stored on the storage devicesso that processing of the data occurs within the storage node(i.e., without transferring any of the data out of the storage node). In this way, the storage nodeis used as a compute node, thereby removing the need to transfer data out of the storage nodeon which the data is stored and removing any latency associated with such a transfer.

106 106 102 106 102 106 106 Further, an example storage nodemay be connected to one or more cloud storage providers of varying levels (e.g., standard cloud storage or lower-class cloud storage, or both, for example, S3® or GLACIER® storage classes). The storage nodeconnected to one or more cloud storage providers may exercise protocol-level control over the allocated cloud storage space available to it on behalf of the hosts. Such control may be via one or more protocols such as HyperText Transfer Protocol (HTTP), HyperText Transfer Protocol Secure (HTTPS), etc. The storage nodemay further send instructions to run the executable object stored on the cloud storage space on behalf of the hosts. Such control may be via current API protocols (e.g., S3, Azure, Swift, etc.) for managing data on the cloud storage space. In some embodiments, the API protocols may be modified (e.g., extended) to allow for better control of executable objects, as will be discussed further below with respect to subsequent figures. In this way, the modified, or extended, API may avoid collisions with current uses of the API and provide better control over executable objects stored in the cloud storage space. Collisions between current API uses and executable code API uses may result in errors occurring in the storage system. In yet other examples, the storage nodemay instead request the data from the one or more cloud storage providers, and the executable object(s), and run the executable object on the relevant data objects at the storage nodebefore returning the data objects to the one or more cloud storage providers (or caching/storing locally if the data has become more in demand again).

104 114 102 114 114 106 114 106 106 114 102 106 106 102 106 114 114 106 106 114 106 114 106 106 106 102 In addition to storage nodes, the distributed storage systemmay include ancillary systems or devices (e.g., load balancers). For example, in some embodiments, a hostmay initiate a data transaction by providing the transaction to a load balancer. The load balancerselects one or more storage nodesto service the transaction. When more than one alternative is possible, the load balancermay select a particular storage nodebased on any suitable criteria including storage node load, storage node capacity, storage node health, network quality of service factors, and/or other suitable criteria. Upon selecting the storage node(s)to service the transaction, the load balancermay respond to the hostwith a list of the storage nodesor may forward the data transaction to the storage nodes. Additionally, or in the alternative, a hostmay initiate a data transaction by contacting one or more of the storage nodesdirectly rather than contacting the load balancer. In some embodiments, the load balancersmay also inspect execute requests according to embodiments of the present disclosure, identify which storage nodescontain the data identified in the execute request, and which storage nodesdo not. For example, load balancersmay maintain a listing of collections of data (e.g., volumes) and identify which storage nodesstore the data requested in execute instructions. In this way, the load balancersmay reduce the workload of the storage nodesthat do not contain any of the requested data by forwarding the execute instructions to only those storage nodesthat may include the data requested. This operation may alternatively be performed by another entity, such as a storage nodeand/or a host.

102 102 104 104 102 116 104 116 116 116 104 118 116 102 114 106 118 106 106 Turning now to the hosts, a hostincludes any computing resource that is operable to exchange data with the distributed storage systemby providing (initiating) data transactions to the distributed storage system, providing executable object manipulation instructions, and/or providing execute instructions (and receiving any results). In an example embodiment, a hostincludes a host bus adapter (HBA)in communication with the distributed storage system. The HBAprovides an interface for communicating, and in that regard, may conform to any suitable hardware and/or software protocol. In various embodiments, the HBAsinclude serial attached small computer system interface (SCSI), iSCSI, InfiniBand, Fibre Channel, and/or Fibre Channel over Ethernet (FCoE) bus adapters. Other suitable protocols include serial advanced technology attachment (SATA), eSATA, parallel advanced technology attachment (PATA), universal serial bus (USB), and FireWire, or the like. In some examples, the host HBAsare coupled to the distributed storage systemvia a front-end network, which may include any number of wired and/or wireless networks such as a LAN, an Ethernet subnet, a PCI or PCIe subnet, a switched PCIe subnet, a WAN, a MAN, the Internet, or the like. To interact with (e.g., read, write, modify, execute/process, etc.) remote data, the HBAof a hostsends one or more data transactions and/or execute instructions to the load balanceror to a storage nodevia the front-end network. Data transactions may contain fields that encode a command, data (i.e., information read or written by an application), metadata (i.e., information used by a storage system to store, retrieve, or otherwise manipulate the data such as a physical address, a logical address, a current location, data attributes, etc.), and/or any other relevant information. Execute instructions may contain fields that encode data selection (e.g., regular expression, string comparison, etc.), executable code to be run (e.g., the executable object(s) stored on storage node(s)), and/or other information relevant to executing the code on the storage node.

114 106 102 106 102 114 106 102 While the load balancers, storage nodes, and the hostsare referred to as singular entities, a storage nodeor hostmay include any number of computing devices and may range from a single computing system to a system cluster of any size. Accordingly, each load balancer, storage node, and hostincludes at least one computing system, which in turn includes a processor such as a microcontroller or a central processing unit (CPU) operable to perform various computing instructions. The computing system may also include a memory device such as random access memory (RAM); a non-transitory computer-readable storage medium such as a magnetic HDDs, SSD, or an optical memory (e.g., compact disk read-only memory (CD-ROM), digital versatile disk (DVD), Blu-ray disk (BD)); a video controller such as a graphics processing unit (GPU); a communication interface such as an Ethernet interface, a Wi-Fi (IEEE 802.11 or other suitable standard) interface, or any other suitable wired or wireless communication interface; and/or a user input/output (I/O) interface coupled to one or more user I/O devices such as a keyboard, mouse, pointing device, or touchscreen.

104 106 104 104 106 104 106 As described above, the storage systemmay distribute the hosts' data across the storage nodesfor performance reasons as well as redundancy. The distributed storage systemmay be an object-based data system. The storage systemmay be a distributed object store that spans multiple storage nodesand sites. In brief, object-based data systems provide a level of abstraction that allows data of any arbitrary size to be specified by an object identifier. Object-level protocols are similar to file-level protocols in that data is specified via an object identifier that is eventually translated by a computing system into a storage device address. However, objects are more flexible groupings of data and may specify a cluster of data within a file or spread across multiple files. Each object may be stored on single node or may be fragmented and stored across multiple nodes or hosts within the object storage system. Object-level protocols include cloud data management interface (CDMI), SWIFT, and S3. A data object represents any arbitrary unit of data regardless of whether it is organized as an object, a file, or a set of blocks. Further, an executable object represents executable code able to be run by the storage system(e.g., storage node).

106 106 106 106 A client may store content, for example, on the storage nodeor an external service cloud. The term “content” may be used to refer to a “data object” or an “object.” The storage nodemay provide the client with a quicker response time for the content than if the content were stored on the external service cloud. It may be desirable for the client to store content that is frequently accessed and/or content that the client desires to be highly available on the storage node, and one or more executable objects pertaining to that content at the same storage node.

102 104 106 106 112 110 106 102 106 106 106 102 106 102 A hostmay instruct the distributed storage systemto run an executable object, for example, on the same nodeas the data (content) is stored, whether it be on the storage nodeor an external service cloud. The term “executable object” may be used to refer to an object that contains executable code that is designed to run on the processors of the specific storage system on which it is stored (e.g., x86, x64, ARM, etc., whether as part of a storage controllerand/or a processor of a storage device). The executable code may be compiled native machine code, code written in a scripting language (e.g., Python, Ruby, Lua, JavaScript, etc.), and/or bytecode that is executed in a runtime environment (e.g., Java). The “executable object” may further include metadata indicating execute permissions of the executable object. The storage nodeand external service cloud may provide a hostwith a quicker response time by running the executable object locally on the data stored on storage nodeas opposed to transferring the data to a compute node that is external to storage nodeand/or external service cloud. Further, whether the response time is quicker or not, running the executable object locally may significantly reduce the amount of traffic between the storage nodeand host, as data is not pulled from the storagefor processing at the host. This further saves the system from the data transfer burden and potential cost associated therewith.

2 FIG. 200 200 100 200 202 204 214 204 118 214 102 214 202 204 202 104 106 106 106 106 106 206 206 206 206 106 208 208 208 1 2 n 1 2 n 1 n is a schematic diagram of a computing architectureaccording to aspects of the present disclosure. The computing architecturemay be an embodiment of the computing architectureas introduced and discussed above. The computing architectureincludes an object storage system, a network, and clients. The networkmay be similar to the network, which may include any number of wired and/or wireless networks such as a LAN, an Ethernet subnet, a PCI or PCIe subnet, a switched PCIe subnet, a WAN, a MAN, the Internet, or the like. The clientsmay be an embodiment of the hosts. The clientsmay communicate with the object storage systemvia the network. The object storage systemmay be an embodiment of the distributed storage systemand include one or more storage nodes, labeled as storage node, storage, and up to storage node. The storage nodes may also be referred individually or collectively as storage node(s). Each storage nodemay store one or more data objects, labeled as data object, data object, and up to data object. The data objects may also be referred to individually or collectively herein as data object(s). Each storage nodemay further store zero or more executable objects, labeled as executable objectup to executable object. The executable objects may also be referred to individually or collectively herein as executable object(s).

202 106 106 112 112 106 106 206 208 106 206 206 206 208 106 206 206 206 208 208 106 206 206 206 206 202 206 206 106 206 206 206 106 206 206 206 206 206 110 106 2 FIG. 1 1 2 n 1 2 3 4 m 1 n 3 5 6 o 1 3 3 6 3 6 The illustrated object storage systemofincludes three exemplary storage nodes. Each storage nodeincludes one or more storage controllers(e.g., in a high-availability configuration in some examples). In some embodiments, the storage controllersperform the read, write, modify, and execute actions within the storage node. Each storage nodefurther includes storage devices for storing data objectsand executable objects. In the illustrated example, storage nodestores data object, data object, up to data objectand an executable object. Further, illustrated storage nodestores data object, data object, up to data objectand executable objectup to executable object. And storage nodestores data object, data object, up to data object. Each data objectmay represent a whole object stored in the object storage system. As illustrated, each storage nodestores different, unique data objects. However, data may also be replicated across storage nodesfor redundancy purposes such that data objectis the same object as data object. Additionally, or alternatively, data may be split across multiple data objectsand stored on different storage nodes. For example, data objectmay include a first portion of data and data objectmay include a second portion of data where the combination of data objectand data objectrepresents the specific data item. The data objectsmay be stored on one or more storage devicesat their respective storage nodes.

106 208 208 208 106 208 106 202 106 106 208 112 208 110 As illustrated, while storage nodescan store one or more executable objects, some might not store an executable objectat all. Furthermore, each executable objectmay be stored on more than one storage node(e.g., replicated as needed or desired). Each executable objectmay be configured to execute within the storage nodeof the object storage system. That is, the code may be compiled to run on the processor architecture (e.g., x86, x64, ARM, etc.) of the storage nodeor be written in a scripting language (e.g., Python, Ruby, Lua, JavaScript, etc.) that may be interpreted by the storage node. This may include, for example, storing executable code in the executable objectthat is able to be run, or executed, by the storage controller. As another example, this may include executable code in the executable objectthat is able to be run, or executed, by a processor of a storage device.

214 202 208 202 204 208 202 208 206 208 208 202 206 202 The clientsmay communicate with the object storage systemusing a standardized object storage protocol (e.g., S3, Azure, Swift, etc.) or another suitable communication protocol. In some embodiments using the object storage protocol may be extended, or modified, to include the ability to store an executable objectin the object storage system, to allow the clientsto identify the executable object as executable, to control the execution of the executable objecton the object storage system, to control permissions of the executable object, and to identify a set of data objectsfor the executable objectto process. For example, the object storage protocol may be extended (e.g., include additional headers related to the creation, storage, modification, and execution of executable objectswithin the object storage system) to provide a matching expression (e.g., regular expressions) that identifies a subset of data objectsof interest within the object storage system.

206 208 106 206 208 106 106 206 206 206 106 206 106 106 206 106 208 206 106 208 In addition to the data objectsand the executable objects(and the metadata stored as part of each type of object with that object), each storage nodemay also store general metadata associated with the data objectsand the executable objects. For example, each storage nodemay maintain a local index of objects stored on the storage node(an example of general metadata). The index may also include some or all of the immutable metadata (e.g., creation time, etc.) of the data objects. The selection information may include regular expressions, string comparisons, and/or other methods of identifying data objectsof interest based on bucket names, object names, prefixes, tags, and/or other metadata. The selection information may be used to filter the data objectsin the local index. The storage nodemay generate a physical location map based on the filtered data objectsof interest. The storage nodemay further aggregate the physical location map with the physical location maps of other storage nodesto create a cluster-wide view physical location map of all data objectsof interest. The storage nodemay further provide the physical location map(s) to the executable objectsin order to identify the data objectsof interest. In some examples, the storage nodemay use the physical location maps to generate an overlay filesystem view. The filesystem view provides a synthetic hierarchical representation of the flat object address space. An executable objectmay incorporate legacy code that is designed to work with a hierarchical filesystem view.

208 106 106 208 214 208 202 214 202 206 208 208 208 206 206 206 202 106 106 106 106 202 208 106 106 206 106 106 206 206 106 206 106 206 106 106 208 1 1 2 3 1 2 n 1 1 1 2 2 3 n 1 2 1 Executable objectsmay be replicated from one storage nodeto another storge node. In some examples, an executable objectmay be replicated as part of a request from clientto run the executable objecton the object storage system. For example, the clientmay send a request to the object storage systemincluding a matching expression to identify data objectsof interest to process, an identifier of an executable object(e.g., executable object) to run, and instructions to run the executable object. The matching expression may identify data object, data object, and data objectfor processing (as an example). The object storage systemmay provide the request to each of the storage nodes(e.g., storage node, storage node, and up to storage node). The object storage systemmay store the executable objecton each of the storage nodes. Each storage nodemay use the matching expression included in the request to determine whether any data objectsof interest are stored on the storage node. For example, storage nodemay determine data objectand data objectmatch the matching expression, storage nodemay determine that data objectmatches the matching expression, and storage nodemay determine that no data objectsmatch the matching expression. Storage nodeand storage nodemay then run the executable objectaccording to the received instructions.

202 106 106 106 106 208 208 106 206 106 208 106 106 208 1 2 n 1 1 3 2 1 1 2 1 Alternatively, the object storage systemmay forward the request to each of the storage nodes(e.g., storage node, storage node, and up to storage node) but store the executable object(e.g., executable object) on a single storage node (e.g., storage node). After identifying data objectas matching the matching expression, storage nodemay request a copy of the executable objectfrom storage node. Storage nodemay then run the executable objectaccording the received instructions.

106 208 106 206 106 202 208 1 2 2 Alternatively, the storage nodein this example may determine from one or more physical location maps to push the executable objectto storage nodewhen one or more of the data objectsat storage nodematches the expression. As demonstrated in the above example, the storage systemprovides an improvement over existing pipelining paradigms that rely on moving large amounts of data out of a storage system to a compute system, and back to the storage system using large amounts of network resources. In contrast, the data-centric pipelining described herein moves small applications (e.g., executable objects) into a storage system to process the data in place. Such in-place data-centric pipelining decreases the amount of network resources used and may increase the speed of processing the data. Aspects of the present disclosure are discussed below with respect to the additional figures.

3 FIG. 1 2 FIGS.and 300 300 214 202 106 106 300 300 1 2 Turning now to, illustrated is an exemplary process flowfor running executable objects in an object storage system according to some embodiments of the present disclosure. Process flowillustrates a flow between a client, a storage system, a storage node, and a storage nodeas described above with respect to. It is understood that additional actions can be provided before, during, and after the actions of the process flow, and that some of the actions described can be replaced or eliminated for other embodiments of the process flow.

302 214 202 214 202 118 204 204 214 202 214 202 202 202 202 At action, the clientconnects to the storage system. The clientmay connect to the storage systemvia a network such as networkor networkdescribed above. The clientmay connect using an object storage protocol (e.g., S3, Azure, Swift, etc., whether in an unmodified form, or modified with extensions as discussed herein). The connection may allow the clientaccess to the storage systemincluding read, write, modify, and execute permissions. In some embodiments, the communication from the clientmay be from an application that has been designed to use executable objects running on the storage system. That is, instead of the application requesting the data objects be transferred from the storage systemfor processing, the application transfers executable objects to the storage system, and/or instructions for processing the data objects within the storage system.

304 214 202 208 202 208 106 214 208 208 106 202 208 106 208 106 208 208 208 208 208 202 214 202 208 208 208 At action, the clientsends instructions to the storage systemfor running an executable objectin the storage system. In some examples, this corresponds to when a related executable objectis already at the one or more storage nodeswhich are a target of the instructions from client(e.g., sending the ID of the executable object). In other examples, this corresponds to sending the executable objectitself to the storage node(s)of interest in storage system, and/or instructions for replication or other manipulations of an executable objectat one or more nodes, followed by (or along with) instructions for running the executable objecton each of the one or more nodes. Additionally, or alternatively, the instructions may identify multiple executable objectsto run along with instructions for running the multiple executable objects. In some examples, the instructions may include both parameters for how to run the executable objectand instructions to run the executable object, while in other examples the instructions may simply include instructions to run the executable objectwith default and/or existing parameters. Having connected to the storage system, the clientmay send instructions to the storage systemusing an object storage protocol (e.g., S3, Azure, Swift, etc.). In some embodiments, the object storage protocol may be overloaded in order to run the executable object. For example, the headers identifying a request to modify an object may be overloaded to run an executable objectwhen the ID of an executable objectis provided. In some other embodiments, the object storage protocol may be extended to include headers specifically for running an executable object.

208 208 206 208 206 208 208 106 208 206 214 Using an extended object storage protocol simplifies the process flow and reduces the chances for errors (as opposed to overloading an existing object storage protocol message). The instructions may include one or more parameters including selector information, executable code to be stored as an executable object, and/or the ID to an existing executable object. The selector information may be used to identify data object(s)to be processed by the executable objectwithout knowing in advance the specific data object(s)to be processed. The instructions may include an instruction to store an executable objectthat may be included with the instructions. Alternatively, the instructions may include an object ID that identifies an already existing executable objectat one or more storage nodesto be run. Other information relevant to the running of the executable objectand the processing of the data object(s)may be included in the instructions sent by the client.

306 202 206 106 202 306 106 106 106 306 106 208 106 208 106 208 306 306 304 214 106 a 1 1 1 b 2 2 b a At action, the storage systemforwards the instructions to the storage nodes. In the illustrated embodiment, the instructions are forwarded individually to each storage nodein the storage system. For example, at actionthe instructions are forwarded to storage node. The executable object included in the instructions may be forwarded to storage nodeto be stored on the storage node. At action, the instructions may be forwarded to the storage nodewithout the executable object. As previously discussed, the executable objectincluded in the instructions may not be replicated to each storage nodealong with the instructions. Instead, the object ID of the executable objectand object selector information may be forwarded to the storage node. Alternatively, the executable objectitself may be replicated along with the instructions. While illustrated as distinct actions, actionsandmay alternatively constitute the action, such that the clientcommunicates with the storage node(s)without requiring forwarding first.

202 206 202 106 206 214 In an alternative embodiment, the storage systemmay maintain a cluster-wide index of physical locations of data objects. In such an embodiment, the storage systemmay forward the instructions to those storage nodesthat have stored data objectsidentified by the instructions sent by the client.

308 106 208 106 106 208 206 106 106 206 208 2 1 2 2 2 At action, the storage nodemay optionally request the executable objectfrom the storage node. The determination of whether or not the storage noderequests the executable objectmay be based on whether any of the requested data objectsreside on the storage nodeas determined by the selector information. After receiving the instructions, including the object selector information, the storage nodemay filter its index of locally available objects based on the selector information included in the instructions. The selector information may include criteria such as, for example, regular expressions, string matching, etc. for identifying data objectsto be processed by the executable object. The selector information may incorporate whole or partial bucket names, object names, prefixes, tags, and/or other metadata, to name some examples.

106 206 206 106 208 1 2 For example, the storage nodemay include several buckets named “logs,” “genomes,” and “records.” The selector information may identify the “logs” bucket as the bucket containing the data objectsof interest. The selector information may further identify a subset of the log data based on the object names. For example, each object may be named according to a pattern such as “log_yyyy_mm_dd_hh_mm_ss” where the object name identifies a specific time period covered by the log where “yyyy” is the four digit year, “mm” is the two digit month, “dd” is the two digit day, “hh” is the two digit hours, “mm” is the two digit minutes, and “ss” is the two digit seconds. The logs of interest may be selected using a wildcard or regular expression such as “log_2020_06_15_*” that identifies all objects that include log information from Jun. 15, 2020. If one or more data objectsare identified by the selector information on the storage nodethen the request for the executable objectmay be sent.

310 106 208 308 106 208 106 202 208 208 208 1 1 2 3 FIG. At optional action, the storage noderesponds to the optional request for the executable objectfrom optional action. The storage nodesends a copy of the executable objectto the storage node. A design decision may be made as to whether the storage systemoperates by pushing executable objectsto all storage nodes or pulling executable objectsas needed. The illustrated example ofprovides an example of pulling executable objectsas needed/desired.

206 208 202 208 206 106 208 106 208 206 106 208 208 202 214 208 Depending on the nature and/or location of the data object(s)to be processed, pulling the executable objectmay be a more efficient use of the storage systemresources than pushing the executable object. For example, in situations where the data object(s)to be processed is concentrated in a small subset of storage nodes, this removes the overhead of copying the executable objectto storage nodesthat may not have need to run the executable object. Conversely, in an example where the data objectsare found on a majority of the storage nodes, pushing the executable objectmay be more efficient to reduce the amount of extra network traffic created by pulling the executable object. In some embodiments, the storage systemmay determine when to use a push or a pull scheme for replicating the executable objects (e.g., such that the system may either determine at one time to use either scheme, or may vary dynamically and automatically (or based on user instruction) between schemes). In some other embodiments, the clientmay identify whether to use a push or a pull scheme for replicating the executable objects.

312 106 208 206 208 214 208 206 208 At action, the storage nodes(all those that have the identified executable objectrelating to the identified data object(s)) run the executable objectas instructed by the client. Running the executable objectmay include processing and/or modifying any of the data objectsidentified by the selector information included in the original instructions. This is an improvement on current pipelining paradigms. Data may be processed using fewer network resources by using the executable objectto provide in-place data processing (e.g., data-centric pipelining).

208 106 202 208 206 312 106 208 106 312 106 208 106 112 106 106 208 112 110 202 208 106 106 206 a 1 1 1 b 2 1 2 1 2 FIGS., The executable objectis run by each respective storage nodein the storage systemthat has the executable objectlocally as well as the identified data objectsaccording to the instructions received. That is, at actionthe storage noderuns the executable objectthat is stored in the storage nodeand at actionthe storage noderuns the executable objectthat is stored in the storage node. In some embodiments, the executable object may be run by storage controllers, such as storage controllers(), located at each storage node. In some other embodiments, an additional compute resource, including at least a processor and RAM, may be included in each storage nodeto run the executable objects, either adjacent to storage controllersand/or as part of one or more storage devices. Additionally, or alternatively, multiple compute resources, including at least a processor and RAM, may be located at the storage systemlevel where each compute resource runs the executable objectsfor one or more storage nodes. Varying levels of compute resources may also be run in some combination, either in parallel or serially, at a given storage nodefor the data objectsat that node.

313 313 106 214 208 313 106 106 208 106 106 208 106 106 214 304 208 106 304 106 106 106 214 106 106 214 a b a 1 1 2 2 1 1 2 2 1 Boxesandillustrate different embodiments for handling communication between the storage nodesand the clientwhile an executable objectis running. Referring to box, the storage nodesare illustrated operating in a collaborative manner. This may refer, for example, to the storage node(i.e., the executable objectrunning on the storage node) and storage node(i.e., the executable objectrunning on the storage node) communicating with each other. The storage nodemay communicate with the clientafter receiving the initial instructionsto run the executable object. In this example, the storage nodemay forward a request (e.g., additional requests that are not the instructions) to the storage node. Similarly, the storage nodemay send any response to the request to storage nodefor sending back to the client. Collaborative communication between the different storage nodesmay significantly reduce the amount of traffic between the storage nodesand the client, and may also result in a quicker response time. Moreover, this saves the system from the data transfer burden and potential cost associated therewith.

314 214 208 106 208 214 208 304 312 214 202 202 214 208 106 208 1 1 1 1 1 1 At action, the clientsends a request to the executable objectrunning on the storage node. The request may include a status update, a request for more information, additional information for the executable object, or any number of requests from the client. Such requests are sent while an executable objectis running, for example responsive to action(running at action). Notably, the request sent by the clientbypasses the storage system. As a result, the request may be sent by a protocol other than object storage protocol (e.g., S3, Azure, Swift, etc.) used to communicate with the storage system. The request may be a communication from an application running on the clientto the executable objectrunning on the storage node(e.g., where the application and executable objecthave been designed to work together to improve the performance of the application).

316 313 208 106 214 208 106 208 106 a 1 1 1 2 1 1 At action, still part of box's example, the executable objectrunning on storage nodeforwards the request from the clientto the executable objectrunning on storage node. Additionally, the executable objectrunning on storage nodemay process the request received in order to respond to it appropriately.

318 208 106 106 313 106 208 106 208 318 106 208 106 214 1 2 2 a 2 1 1 1 2 1 1 At action, the executable objectrunning on the storage noderesponds to the request. The executable object running on the storage nodeprocesses the request and then responds to the request. In a collaborative process, as illustrated by boxthe executable object running on the storage noderesponds to the executable objectrunning on the storage node. In other words, the executable objectsends its response at actionfrom the storage nodeto the executable objectrunning at storage node, instead of directly back to the client.

320 106 214 208 106 208 106 214 214 1 1 1 1 2 At action, the executable object running on the storage noderesponds to the original request from the clientwith the requested information. The executable objectrunning the storage nodeincludes the responses from the executable objectsrunning on the other storage node(as a singular example, which may be scaled to any number of nodes) when responding to the clientwith the requested information (e.g., status update information, confirmation that the additional information was received and incorporated, etc.) when sending the response back to the client.

313 106 214 313 214 106 106 106 106 214 204 106 106 313 106 106 208 214 206 b 1 1 2 2 1 2 b a Referring to the box, the storage nodesare illustrated as communicating directly with the client. The aspects illustrated in boxmay refer to situations where the clientcommunicates with the storage node(i.e., the executable object running the storage node) and with the storage node(i.e., the executable object running on the storage node). For example, the clientmay send a request (i.e., an additional request that is not the instructions) separately and individually to both the storage nodeand the storage node. This is in contrast to the collaborative example described above with respect to box, where a request to one of the storage nodesis conveyed via another storage node. In some situations, this may result in a faster response from the executable objectsuch as when the clientis addressing a single data object.

322 214 106 208 314 202 1 At action, the clientsends a request to the executable object running on the storage node. Again, examples of requests may include status updates, requests for more information, additional information, etc., targeting executable object(s)that are already running. As discussed above with respect to action, the request may be sent by a protocol other than the object storage protocol (e.g., S3, Azure, Swift, etc.) used to communicate with the storage system.

324 208 106 214 318 320 1 1 At action, the executable objectrunning on the storage nodeprocesses the request (e.g., as part of running its executable code for its intended purpose), and responds to the client, as has been previously described with respect to actionsand.

326 214 106 2 At action, the clientseparately sends a request to the executable object running on the storage node, as has been previously described.

328 208 106 214 313 208 106 106 322 324 1 2 b 1 1 2 At action, the executable objectrunning on the storage nodeprocesses the request and responds to the clientwith the requested information (e.g., status updates, requests for more information, additional information, etc.). With respect to box, this may refer to each of the executable objectsat each of nodesandresponding to the separate requests received at actionsand, respectively.

4 FIG. 4 FIG. 1 2 FIGS.and 400 106 202 400 112 106 106 400 400 Having discussed the general process flow,illustrates a flow diagram of running an executable object on a storage node according to embodiments of the present disclosure. In the description of, reference is made to elements offor simplicity of illustration. In an embodiment, the methodmay be implemented by an exemplary storage nodethat is part of a larger object storage system such as, for example, object storage system. In another embodiment, the methodmay be implemented by an exemplary storage controllerthat is part of a storage node. The description below will describe this with respect to a given storage node. It is understood that additional steps can be provided before, during, and after the steps of the method, and that some of the steps described can be replaced or eliminated for other embodiments of the method.

402 106 208 214 202 106 214 208 202 208 106 208 106 106 2 FIG. At block, the storage nodereceives instructions to run an executable object (e.g. executable objectof), from a client. The instructions may be received via the object storage system, via another storage node, or directly from a clientdescribed above. The instructions may be received using an object storage protocol such as, for example, S3, Azure, Swift, etc. In some embodiments, the object storage protocol may be extended to include additional headers designed specifically for handling executable objectson the object storage system. The instructions may include selector information, an executable code to be stored as an executable object, and/or an ID of an executable object. The selector information may be used to filter objects stored on the storage nodeto identify data to be processed by the executable object. The executable code stored in the executable object may be configured to run on the storage node. For example, the code may be compiled to run on processors of the storage node (e.g., x86, x64, ARM, etc.). As another example, the code may be written in a scripting language (e.g., Python, Ruby, Lua, JavaScript, etc.) that is interpreted and run on the processor of the storage node.

404 106 206 106 206 404 402 206 At block, the storage nodefilters the data objectsstored in it according to the received instructions. As noted previously, each storage nodemay maintain an index of locally available data objectsand/or data object fragments (e.g., a multi-part upload, Erasure Code, etc.). For example, a large data object may be stored across multiple nodes where a first portion of the data is stored as a first object on a first node, a second portion of the data is stored as a second object on a second node, and a parity object (e.g., Erasure Code) is stored on a third node. The object fragment boundaries (e.g., first object and second object) may be defined by the user or by the object storage system as part of storing the data. The index may be filtered at blockaccording to the selector information provided in the received instructions at block. The selector information may provide information for identifying data objectsof interest. Some examples of information include regular expressions, string comparisons, wildcards, etc. The selector information may incorporate bucket names, object names, prefixes, tags, and/or other metadata. That is, the data objects may be named according to a defined scheme in order to more easily find the data objects of interest. For example, data objects containing log data may be named “logs_yyyy_mm_dd” where “yyyy_mm_dd” identifies the year, month, and day that the logs were generated. Furthermore, the log objects may be stored in buckets named “logs” or “logs_yyyy” that allow for filtering on a larger scale (as an example).

406 106 206 404 206 400 408 400 106 402 206 106 208 208 206 400 410 At decision block, the storage nodedetermines whether any data objectswere identified by the filtering done at block. If it is determined that no data objectswere identified by the filtering, then the methodproceeds to blockwhere the methodends with respect to the storage nodeand the request received at block. That is, with no data objectsidentified as matching the selector information, the storage nodedoes not run the executable objectas instructed, since there is no data at that node on which the executable objectis intended to run. If, instead, it is determined that one or more data objectswere identified by the filtering, then the methodproceeds to the decision block.

410 106 208 106 106 208 208 106 208 208 106 400 412 At decision block, the storage nodedetermines whether the requested executable objectis available (i.e., stored) on the storage node. As previously mentioned, the executable code may be included in the received instructions to be stored on the storage nodeas an executable object. Alternatively, the executable objectmay already be stored on the storage node. However, in some examples the executable objectmay not be included in the instructions or already be stored on the storage node. If it is determined that that the executable objectis not stored on the storage node, then the methodproceeds to block.

412 106 208 106 106 402 208 106 106 106 106 106 106 106 106 114 106 202 208 106 208 414 1 2 2 FIG. At block, the storage nodepulls the executable objectfrom another storage node. For example, the storage nodemay use the executable object ID included in the instructions received at blockto request the executable objectfrom another storage node. The other storage nodemay be the storage nodethat forwarded the instructions to the storage node(e.g., nodesandof). Alternatively, the other storage nodemay be identified by a general query of storage nodesand/or the storage system controller, such as the load balancer. As another example, the other storage nodemay be identified by an aggregated map with a larger view, such as a cluster-wide view, of the system. After sending the request for the executable object, the storage nodereceives a copy of the executable objectand stores it. The method then proceeds to block.

410 208 106 400 414 If, instead, at decision blockit is determined that the executable objectis stored on the storage nodealready, then the methodproceeds to block.

410 412 414 106 206 404 202 208 From either of blocksand, at blockthe storage nodegenerates a synthetic filesystem view of the data objectsidentified at block. This may include converting the flat address space of the object storage systemto a hierarchical overlay that can be used to find the physical location of the objects. In some examples, this may include using characters in the object name, the bucket name, and/or metadata as delimiters to form a synthetic hierarchical view of the objects and define the directories. This may group similar objects into directories of the synthetic hierarchical view based on a naming convention (e.g., object name and/or bucket name) and/or metadata. For example, two objects named “log_2020-03-15_stationA” and “log_2020-03-15_stationB” may be stored in a bucket named “logs.” The synthetic hierarchical view may be formed such that the first object is translated into a directory structure “logs/log/2020-03-15/stationA.txt” and the second object is translated into a directory structure “logs/log/2020-03-15/stationB.txt.” The executable objectmay then use the synthetic hierarchical view to find the first and second objects for processing as discussed further below. Alternatively, other characteristics of the objects may be used to define the directories of the synthetic hierarchical filesystem.

208 206 208 206 208 208 208 414 414 The synthetic filesystem view may assist the executable objectin addressing and processing the identified data objects. For example, the synthetic filesystem view may enable the executable objectto use kernel level translations to view data of the data objectswithout expending the resources typically used with standard object access protocols. This may be useful, for example, when the executable objectincludes legacy code (e.g., legacy code such as file access code designed to be used with a filesystem) that is designed for use with a traditional hierarchical filesystem; the legacy code may be reused in the executable objectwith the use of the synthetic filesystem view. Where legacy code is not included as part of the executable object, generating a filesystem view may not be necessary such that blockmay be optional. However, even if not necessary, generating a filesystem view may still be performed at block.

416 106 208 106 112 106 106 106 106 106 106 106 416 At block, the storage nodemeasures available resources in preparation to run the executable object. Resources available on the storage nodemay include compute resources (e.g., processors) and memory resources (e.g., RAM, HDD, SDD, etc.), network resources (e.g., throughput), etc. In some embodiments, the compute resources that run the executable objects may be in the storage controllers. While the storage nodemay use these resources heavily during the storage and retrieval of data objects for clients, for many object storage applications, especially after loading data into the storage nodeis complete, the compute and memory resources may remain largely unused. Additionally, the compute resources of storages nodesthat are close to or at capacity may be largely unused as new data is not being written to the storage nodeand reading data uses minimal resources. The storage nodemeasures these resources to ensure that the primary functions of data storage and retrieval might not be interrupted. When monitoring compute resources, the storage nodemay monitor the current and historical processor load used for objects storage tasks (e.g., read, write, modify, etc.) as well as object execute tasks (e.g., running executable objects). The storage nodemay store historical use data such as, for example, average workload, average idle time, peak workload, peak idle time, and/or time windows associated with each of those, just to name a few examples. For example, processor workloads may be the highest during normal work hours. Alternatively, the processor workload may be spread across the day when storing log file information. In some examples, a system administrator or other user may choose to deactivate the resource check at block(or this may otherwise be an optional action).

418 106 208 416 106 208 106 106 106 208 At decision block, the storage nodedetermines whether there are sufficient available resources to run the executable object, based on the information obtained/determined from block. The storage nodemay use a heuristic to determine whether there are sufficient resources. That is, even if there are available resources to run the executable object, the storage nodemay determine that there are not sufficient resources if there is not a resource buffer to account for an uptick in resource use for the primary objectives of storing and/or retrieving data. Alternatively, the storage nodemay determine that there are available resources whenever there is any spare capacity. In such a scenario, the storage nodemay run the executable object, but may throttle the speed of execution to not impact the performance of object storage and retrieval.

208 400 416 400 420 If it is determined that there are not enough available resources to run the executable object, the methodreturns to blockto measure the available resources until there are available resources. If, instead, it is determined that there are available resources, then the methodproceeds to block.

420 106 208 208 206 206 106 206 404 208 106 106 208 106 208 208 416 At block, the storage noderuns the executable object. The executable objectmay process the data in the identified data objectswithout transferring the data objectsout of the storage node(i.e., bringing the compute action as close to the data objectsas possible). This may be referred to as in-place data-centric pipelining. Returning to the example of log files from blockfor sake of a simple illustration, the executable objectmay process the log files and identify any anomalies and/or errors of interest in less time than was traditionally possible with object storage by processing the data files locally in the storage node. Further, by processing at the storage node, there are fewer resources used transferring the data to a compute node (since transfer is not performed) and the overall process may be faster without the time requirements of transferring large amounts of data. The executable objectmay run to completion, at which point the storage nodemay return to handling data storage requests and waiting for instructions to run another executable object(or, alternatively, the running of the executable objectmay occur concurrent to handling data storage requests generally, based on availability of resources such as noted above with respect to block).

5 FIG. 5 FIG. 1 2 FIGS.and 4 FIG. 500 106 202 500 112 106 202 106 106 500 500 illustrates a flow diagram of running an executable object on more than one storage node according to embodiments of the present disclosure. In the description of, reference is made to elements offor simplicity of illustration. In an embodiment, the methodmay be implemented by exemplary storage nodesthat are part of a larger object storage system such as, for example, object storage system. In another embodiment, the methodmay be implemented by exemplary storage controllersthat are part of storage nodes. The description below will describe this with respect to the storage systemand a group of storage nodesas compared to the single storage nodeimplementation example of. It is understood that additional steps can be provided before, during, and after the steps of the method, and that some of the steps described can be replaced or eliminated for other embodiments of the method.

502 202 208 214 402 208 208 202 2 FIG. 4 FIG. At block, the object storage systemreceives instructions for running an executable object (e.g., executable objectof), from a client, similar to blockof. The instructions may include criteria for identifying data objects to process, such as selector information (e.g., regular expression, wildcards, string comparison, etc.). The instructions may also include executable code to be stored in an executable objectand/or the object ID of an executable object to be used to process the identified data. The instructions may be received using an object storage protocol such as, for example, S3, Azure, Swift, etc. In some embodiments, the object storage protocol may be extended to include additional headers designed specifically for handling executable objectson the object storage system.

504 202 106 208 106 206 106 208 202 106 206 106 106 202 106 202 206 At block, the object storage systemidentifies which storage nodeson which to run the executable object. Storage nodesthat store one or more data objectsidentified by the execute instructions are part of a group of storage nodesthat will run the executable object. In some embodiments, the storage systemmay identify which nodesstore data objectsbased on a cluster-wide index of data object locations. The cluster-wide index may include information such as bucket names, object names, prefixes, tags, and/or other metadata. The execute instructions may then be forwarded to the identified storage nodes. In some other embodiments, storage nodeswithin the object storage systemmay receive the instructions and may maintain their own index of object locations. The local storage node index may include bucket names, object names, prefixes, tags, and/or other metadata. The storage nodesmay receive the instructions from the object storage systemand filter their own index to identify one or more data objectsbased on the instructions.

506 202 208 106 208 106 208 106 208 106 208 106 202 106 208 206 208 208 106 202 At block, the object storage systemreplicates the executable objectto the identified storage nodes(e.g., when the executable objectis not already present at the given storage nodes). As previously discussed, the executable objectsmay be replicated to the different storage nodesusing either a push or a pull paradigm. When using a push paradigm, the executable objectthat is included in the received instructions or that is identified by the received instructions is pushed to the different storage nodesalong with the instructions. In one example, the executable objectand instructions are pushed to every storage nodein the object storage system, regardless of whether every storage nodeneeds the executable objectand/or has data objectsrelevant to that executable object. In another example, the executable objectand the instructions are pushed to only those storage nodesidentified by the object storage systemas storing data objects identified by the received instructions (e.g., identified by the cluster-wide index).

106 202 208 502 208 106 206 106 206 206 106 106 208 106 208 106 106 208 208 106 502 106 2 FIG. 1 n 1 1 2 1 1 2 2 When using the pull paradigm, the instructions are sent to the different storage nodeswithin the object storage systemalong with the object ID of the executable objectidentified in block, instead of the executable objectitself. In this example, each storage nodemay use the instructions so received to determine whether data objectsidentified by the instructions are stored on the respective storage node. By way of example and reference to, if one or more data objects-identified by the instructions are stored on the storage node, then the storage noderequests (i.e., pulls) the executable objectfrom a storage node. This may correspond to a situation where the executable objectis not already stored on the storage node. In this particular example, the storage nodemay pull the executable objectbecause the executable objectis already at the storage, or was received as part of blockat the storage node.

508 106 206 502 208 106 208 106 208 106 208 208 106 106 208 206 106 1 2 2 FIG. At block, the different storage nodesidentified as storing data objectsidentified by the instructions (received at block) begin the process of running the executable object. In one embodiment, the storage nodesmay run the executable objectin parallel. That is, each storage noderuns the executable objectat the same time (or overlapping times), storage node resources permitting. In another embodiment, the storage nodesmay run the executable objectserially. That is, the executable objectmay run on a first storage node(to give a specific example with reference toagain) to completion before running on a second storage node. Running executable objectsserially on multiple different nodes may be used when the content of a data objectis stored across multiple storage nodes, such as with object data fragments associated with multi-part uploads.

208 106 208 208 508 106 106 500 The following discussion will be used to further illustrate this concept. The principles discussed below may be equally applied to running executable objectsin parallel on different storage nodesexcept that the executable objectsmay be running concurrently. Therefore, in this discussion related to serially running the executable object, at blocka first storage nodeis selected. A different storage nodemay be selected for each iteration of the method.

510 106 416 106 106 106 202 114 106 106 208 106 106 208 416 510 4 FIG. At block, the selected storage nodemonitors its resource use, such as discussed above with respect to blockof. The selected storage nodemay monitor resources at regular intervals so as to minimize the effect of the monitoring on the resources. In other examples, monitoring may be done continually. When monitoring compute resources, the storage nodemay monitor the current and historical processor load used for objects storage tasks (e.g., read, write, modify, etc.) as well as object execute tasks (e.g., running executable objects). The storage nodemay store historical use data such as, for example, average workload, average idle time, peak workload, peak idle time, and/or time windows associated with each of those, just to name a few examples. For example, processor workloads may be the highest during normal work hours. Alternatively, the processor workload may be spread across the day when storing log file information. The object storage system, as an example the load balancer, may minimize the difference between each of the above examples in order to keep the processor workload mostly consistent across storage nodes. Because of this, storage nodesmay generally have idle processor time that can be used to run executable objectson the given storage node. The storage nodemay also monitor memory resources to ensure that there is sufficient memory to run the executable object. As noted at blockas well, blockmay be an optional action.

512 106 208 106 500 510 106 500 514 510 500 508 514 At decision block, the selected storage nodedetermines whether there are sufficient resources to run the executable object. The decision may be made based on the current resource load of the selected storage node. Additionally, or alternatively, the decision may be made based on a time window in order to average the resource use to more accurately determine resource availability. If it is determined that there are not sufficient available resources, then the methodreturns to blockto continue monitoring resources while the selected storage nodeperforms its object storage tasks (e.g., ready, write, modify, etc.). If, instead, it is determined that there are sufficient available resources, the methodproceeds to block(and, if blockis optional and not included, the methodproceeds from blockto block).

514 106 208 208 206 208 208 208 106 106 208 106 208 106 208 208 106 106 106 At block, the selected storage noderuns the executable object. The executable objectmay process the identified data objectswhile it is running, according to the operations specified in the executable code of the executable object. The executable objectmay also communicate with a client-side application while it is running. In some embodiments, the executable objectis given discrete processor time slices to run that are interleaved with object storage processing so as to not entirely interrupt the object storage tasks of the selected storage node. For example, the selected storage nodemay run the executable objectfor a first time slice. The selected storage nodemay then perform object storage tasks (e.g., read, write, etc.) for a second time slice while the executable object has not finished processing data objects, effectively pausing the executable object. The selected storage nodemay then resume running the executable objectfor a third time slice. As another example, both running of the executable objectand other object storage tasks may occur concurrently to each other (e.g., where system resources are sufficient to allow it). In some other embodiments, the selected storage nodemay include multiple processors and/or cores where a first portion of the processors and/or cores may be dedicated to object storage tasks and a second portion of the processors and/or cores may be dedicated to running executable objects. Dividing the processors and/or cores this way may provide predictable performance for the executable objects without degrading the performance of the object storage tasks. In some examples, each storage nodemay have between 8 and 80 processors and/or cores (as a non-limiting example). In yet other embodiments, the selected storage node's multiple processors and/or cores may dynamically assign some portion of the processors and/or cores to object storage tasks, and another portion to running executable objects, and may adjust over time how many processors and/or cores are assigned to each portion depending on a variety of factors, such as current system load and/or user request (to name just a few examples). Further, some combination of a dedicated number of processors and/or cores and dynamically assigned processors and/or cores (e.g., a minimum number specified as dedicated to provide a baseline of predictable performance for executable objects or object storage tasks, with more dynamically assigned when available or reassigned if not).

516 106 208 208 106 106 106 106 208 106 106 208 106 208 208 208 106 500 518 106 500 508 106 106 106 At decision block, the selected storage nodedetermines whether the executable object's task is complete. After the executable objecthas finished its task, the selected storage nodemay determine that the task is complete with respect to that node. The selected storage nodemay further determine whether there are any other storage nodesthat are suspended and waiting for the selected storage nodeto complete processing of the executable object. When executing serially, only one of the storage nodes, the selected storage node, is in an active execution state with respect to an executable objectwhile the remaining storage nodesare suspended for that executable object(or another executable objectthat relies upon a result of the first executable objectcompleting). If it is determined that the job is complete and that all parts of the data objects across the one or more storage nodeshave been processed, the methodproceeds to blockand ends. If, instead, it is determined not all parts of the identified data objects have been processed and/or that one or more storage nodesremain to run, then the methodreturns to blockto select the next storage node. The currently selected storage nodemay be suspended or terminated and the active execution state may be passed to the next selected storage node.

6 FIG. 6 FIG. 1 2 FIGS.and 600 214 202 600 600 illustrates a flow diagram of running a client-side application including an executable object component running on a storage node according to embodiments of the present disclosure. In the description of, reference is made to elements offor simplicity of illustration. In an embodiment, the methodmay be implemented by an exemplary clientthat is communicating with an object storage system such as, for example, object storage system. It is understood that additional steps can be provided before, during, and after the steps of the method, and that some of the steps described can be replaced or eliminated for other embodiments of the method.

602 214 202 214 214 118 204 At block, the clientconnects to the object storage system. The clientmay use an object storage protocol (e.g., S3, Azure, Swift, etc.) to communicate with the object storage system. The object storage protocol may be unmodified, or modified/extended as noted elsewhere herein. The clientmay connect to the object storage system over a network such as networkor networkdescribed above.

604 204 208 202 204 202 208 214 208 202 208 202 208 206 206 208 202 214 208 208 202 106 112 206 208 202 At block, the clientmay store an executable objecton the object storage system. That is, the clientmay send an instruction to the object storage systemto store the executable object. The clientmay use the object storage protocol to store executable code as the executable objectwithin the object storage system. In some embodiments, the object storage protocol may be extended to include additional headers related to the creation, storage, modification, and execution of executable objectswithin the object storage system. For example, a new header may be added that identifies the object to be stored as an executable objectas opposed to a data object. A data objectincludes data that is read, written, and/or modified. In contrast, an executable objectincludes executable code that is intended to be run by the object storage system. As another example, a new header may be added that allows the clientto modify the metadata of the executable objectto grant it execution rights and permissions to execute. As another example, a new header may be added that identifies an executable objectto be run by the object storage system(e.g., by a storage node, storage controller, etc.). Although it is not necessary to extend the object storage protocol, doing so may reduce the chances for errors to occur in distinguishing between data objectsand executable objectsand thereby improve the reliability of the object storage system.

606 204 202 208 208 208 206 208 206 At block, the clientsends instructions to the object storage systemto run the executable object. The instructions may be sent using the object storage protocol. The instructions may identify which executable object(or plural objects) to run. The instructions may further identify a set of data objectsto be processed by the by the executable object. For example, the instructions may include a regular expression, a string for comparison, or another way to identify a set of data objects.

608 204 208 208 208 202 214 208 At block, the clientmay send a request to the executable objectwhile the executable objectis running. Once the executable objectis running on the object storage system, the clientmay communicate directly with the executable objectwithout using the object storage protocol. This may be accomplished, for example, by using an application specific API designed to handle the communication between the client-side application and the executable objects. Some examples may be implemented using HTTP, JSON, XML, or another RESTful API.

610 214 208 208 214 202 At block, the clientreceives a response from the executable object. The response may be sent from the executable objectto the client, without going through other elements of the object storage system. The response may be sent using a protocol other than the object storage protocol.

612 214 208 600 608 214 208 214 208 600 614 At decision block, the clientdetermines whether the task being performed by the executable objectis complete. If it is determined that the task is not complete, the methodreturns to blockwhere the clientmay communicate with the executable object(if the clientdesires any information such as a status update while the executable objectis still running). If, instead, it is determined that the task is complete, the methodthen proceeds to block.

614 208 214 208 214 208 202 214 208 202 208 214 208 202 202 At block, the processing task being performed by the executable objectis complete. While the processing task may be complete, the clientmay continue running the client-side application that had requested the processing by the executable object. The clientmay determine whether or not to terminate the executable objectrunning on the storage system. For example, the clientmay decide to leave the executable objectrunning in an idle state on the object storage system(e.g., only listening for further requests) in anticipation of requesting further information from the executable object. Alternatively, the clientmay decide to send a request to the executable objectrunning on the object storage systemto shutdown in order to free resources of the object storage system.

600 202 A continuous security monitoring tool may be an exemplary implementation of the methodand of executable objects more generally. The continuous security monitoring tool, referred to as “the application” below, may store data in an object storage system, such as the object storage system. The stored data may include computer logs, network logs, business operation data, etc. The object storage system may include local and cloud storage nodes. The data may be stored across one or more local and/or cloud storage nodes of the object storage platform.

214 As the data is generated it may be stored in the object storage system by one or more clients. The data may be stored according to a naming scheme. For example, computer logs may be stored in a bucket named “computer logs” and each object may be named according to a naming scheme of “cl_yyyy_mm_dd” where each object represents the logs for a specific day identified by the four digit year, the two digit month, and the two digit day. The network logs may be stored in a bucket named “network logs” and each object may be named according to a naming scheme of “nl_yyyy_mm_dd” where each object represents the logs for a specific day identified by the four digit year, the two digit month, and the two digit day. The business operation data may be stored in a bucket named “op data” and each object may be named according to a naming scheme of “od_div_yyyy_ww” where each object represents the data of a specific division identified by “div” and by time frame such as a four digit year and a two digit week. Each storage node may store portions of one or more buckets. Each bucket may be stored on one or more storage nodes. Each object may be stored entirely or partially within one or more storage nodes.

In order to derive value from the data (e.g., access, process, use, etc.) the continuous security monitoring tool may have, in the past, had to copy the data objects from the object storage system to one or more compute nodes for processing. The compute node may be as simple as a user's desktop computer or as complicated as a cloud-based processing system. Copying all of the objects to the one or more compute nodes may use a large number of resources including computing resources, network resources, time, money, etc.

An improved way to derive value from the data stored in the object storage system is to use the object storage system as the compute nodes using executable objects according to embodiments of the present disclosure described above (and further below in the examples). Doing so allows the application to process the data in place on the object storage system without the expense (e.g., cost and resources) of transferring large amounts of data out of the object storage system for processing. The executable objects may contain code designed to be run on the object storage system (e.g., hardware, scripting, virtual machine, etc.).

In an exemplary implementation, the application may store the code as an executable object on the object storage system. The application may send instructions to the object storage system to run the executable object. The instructions may also include selector information for identifying the set of data objects of interest. For example, the data objects of interest may be computer log objects. The selector information would then identify only those objects stored in the “computer log” bucket(s). As another example, the data objects of interest may be network log objects from a specific year, such as 2019. The selector information would then identify only those objects in the “network logs” bucket(s) that are named “nl_2019_*”. As another example, the objects of interest may be all objects from a specific year and month, such as March of 2019. The selector information would then identify only those objects that are named “*_2019_03_*” or match the regular expression “/2019_(09|1[0-3])/” to identify objects named for March of 2019 that are named with the weeks of “08”-“13” of the year 2019, which are the week numbers that overlap with March of 2019. Each storage node that includes the data objects of interest may then run the executable object. At this point, the application may communicate directly with the one or more executable objects running on the storage nodes of the object storage system. Then, instead of copying all of the objects from the object storage system, the executable objects may return the information requested by the application, thereby improving the overall efficiency and performance of the different systems involved in the process (e.g., client, storage, compute, etc.).

600 A genomics application may be another exemplary implementation of the methodand executable objects more generally. An exemplary genomics application may store gene sequencing data to an object storage system. Similar to the continuous security monitoring tool, the genomics application may store the data in buckets according to a naming scheme and may name the objects containing the data according to a naming scheme. The naming scheme may be used for identifying objects to be processed by the genomics application. Each object may also include metadata for identifying markers included in the data that is stored in each of the objects. In this example, the metadata is set at the time that the data is stored in the object storage system. Genomics data can include exabytes of information about gene sequences that is built up over time.

There arise instances when researchers learn that they need to look for new markers in the already stored data and update the metadata. One approach to this task is to copy the data out of the object storage system, scan the data for the specific marker sequence, update the metadata, and store the updated objects back to the object storage system. The amount of time and resources to transfer, analyze, update, and transfer even a subset of exabytes of data is very large. With object storage systems charging for data to be transferred from the object storage system, the cost of performing these large transfers may also be factored into the resources required to perform this update. Alternatively, executable objects may be implemented that work with the genomics application to run on the object storage system. By doing so, the executable objects may analyze and update the data in place on the object storage system without the need to perform large data transfers from the object storage system and back to the object storage system. Therefore, the overall efficiency of the process is improved and the cost of performing the analysis is decreased by not having to pay to transfer all of the data out of the object storage system.

The foregoing outlines features of several examples so that those skilled in the art may better understand the aspects of the present disclosure. Those skilled in the art should appreciate that they may readily use the present disclosure as a basis for designing or modifying other processes and structures for carrying out the same purposes and/or achieving the same advantages of the examples introduced herein. Those skilled in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that they may make various changes, substitutions, and alterations herein without departing from the spirit and scope of the present disclosure.