Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. An apparatus, comprising: a memory; and at least one processing device, coupled to the memory, operative to implement the following steps: communicating, by a compute node over a network, with (i) a cluster file system on a first storage tier, and (ii) a second storage tier, wherein the first and second storage tiers comprise respective disjoint subsets of storage, wherein said first storage tier has a substantially lower access time than the second storage tier; providing a client, on said compute node, of said cluster file system with access to one or more files in said cluster file system on said first storage tier using a first object protocol; and executing, by said compute node, a translation shim that converts between the first object protocol and a different, second object protocol corresponding to said second storage tier to provide said client of said cluster file system with one or more of direct read access and direct write access to one or more files on said second storage tier using the first object protocol, wherein said translation shim provides said client of said cluster file system with direct write access to a given file on said second storage tier by at least storing, in a bucket on said second storage tier, (i) a layout for said given file, wherein the layout comprises at least one parameter indicating a location of said given file within said cluster file system on said first storage tier, and (ii) a plurality of shard objects of said given file.
2. The apparatus of claim 1 , wherein said layout for said given file is obtained from a metadata warehouse corresponding to said cluster file system on said first storage tier, and wherein said layout is stored as an object in said bucket on said second storage tier.
This invention relates to a data storage system that manages file layouts across multiple storage tiers. The system addresses the challenge of efficiently storing and retrieving file layouts in a distributed storage environment, particularly when dealing with large-scale data processing. The apparatus includes a first storage tier and a second storage tier, where the first tier is optimized for high-performance access, while the second tier provides cost-effective, scalable storage. The system retrieves the layout of a given file from a metadata warehouse associated with a cluster file system on the first storage tier. This layout defines the storage structure and access patterns for the file. The layout is then stored as an object in a bucket on the second storage tier, enabling efficient retrieval and management of file metadata across the storage tiers. The apparatus ensures that file layouts are readily available for data processing tasks while optimizing storage costs by leveraging the second tier for metadata storage. This approach improves performance by reducing the need to frequently access the first storage tier for metadata, while maintaining scalability and cost efficiency. The system is particularly useful in environments where large volumes of data are processed, such as in big data analytics or distributed computing systems.
3. The apparatus of claim 1 , wherein said direct read access of said given file from said second storage tier comprises said translation shim obtaining storage location information from a metadata warehouse, obtaining contents of said bucket on said second storage tier storing said layout for said given file and said plurality of shard objects of said given file and providing shard object identifiers on said second storage tier to said client of said cluster file system; and said cluster file system reading one or more of said plurality of shard objects of said given file from said second storage tier using said shard object identifiers, wherein said translation shim transforms object get operations from said second storage tier to file block read input/output operations.
4. The apparatus of claim 1 , wherein, in response to said client of said cluster file system opening said given file with a predefined attribute, said translation shim selects a substantially fastest tier that stores said given file when said given file is stored on a plurality of storage tiers.
5. The apparatus of claim 4 , wherein said translation shim obtains said layout for said given file and storage address information for said given file from said bucket on said second storage tier corresponding to said given file.
6. The apparatus of claim 1 , wherein said translation shim is further configured to detect a request by said client of said cluster file system to archive said given file, and provide a data movement command to a copy agent of a Hierarchical Storage Manager on said cluster file system to archive said given file on said second storage tier.
7. The apparatus of claim 6 , wherein said translation shim is further configured to translate one or more protocols between said first storage tier and said second storage tier during a Hierarchical Storage Management operation.
8. The apparatus of claim 6 , wherein said translation shim is further configured to one or more of provide a request to delete said given file to a metadata server on said cluster file system and update a metadata warehouse with one or more locations of said given file on said second storage tier.
9. The apparatus of claim 1 , wherein said second storage tier comprises an object store capacity tier, and wherein said client of said cluster file system is provided one or more of said direct read access and said direct write access to said one or more files on said second storage tier using the first object protocol based at least in part on an internet protocol (IP) address associated with said object store capacity tier.
10. The apparatus of claim 1 , further comprising the steps of intercepting a file lookup request for a file from a file system client and providing said client with location information for copies of said file on one or more of said first storage tier and said second storage tier, and wherein said file system client performs an input/output operation on said file using said location information.
11. A method, comprising the steps of: communicating, by a compute node, over a network with (i) a cluster file system on a first storage tier, and (ii) a second storage tier, wherein the first and second storage tiers comprise respective disjoint subsets of storage, wherein said first storage tier has a substantially lower access time than the second storage tier; providing a client, on said compute node, of said cluster file system with access to one or more files in said cluster file system on said first storage tier using a first object protocol; and executing a translation shim that converts between the first object protocol and a different, second object protocol corresponding to said second storage tier to provide said client of said cluster file system with one or more of direct read access and direct write access to one or more files on said second storage tier using the first object protocol, wherein said translation shim provides said client of said cluster file system with direct write access to a given file on said second storage tier by at least storing, in a bucket on said second storage tier, (i) a layout for said given file, wherein the layout comprises at least one parameter indicating a location of said given file within said cluster file system on said first storage tier, and (ii) a plurality of shard objects of said given file.
12. The method of claim 11 , wherein said layout for said given file is obtained from a metadata warehouse corresponding to said cluster file system on said first storage tier, and wherein said layout is stored as an object in said bucket on said second storage tier.
A method for managing file layouts in a distributed storage system involves retrieving a file layout from a metadata warehouse associated with a cluster file system on a first storage tier and storing the layout as an object in a bucket on a second storage tier. The system includes multiple storage tiers, where the first tier is a high-performance cluster file system and the second tier is an object storage system. The method ensures efficient file access by maintaining metadata about file layouts in a centralized metadata warehouse, which allows for quick retrieval and distribution of layout information. The layout data, which defines how a file is distributed across storage nodes, is then stored as an object in a bucket on the second storage tier, enabling scalable and durable storage of this metadata. This approach improves performance by reducing the need to recompute or fetch layout information repeatedly, while also leveraging the cost-effective and scalable nature of object storage for metadata management. The method is particularly useful in large-scale distributed systems where files are frequently accessed and modified, ensuring that layout information remains consistent and readily available.
13. The method of claim 11 , wherein said direct read access of said given file from said second storage tier comprises said translation shim obtaining storage location information from a metadata warehouse, obtaining contents of said bucket on said second storage tier storing said layout for said given file and said plurality of shard objects of said given file and providing shard object identifiers on said second storage tier to said client of said cluster file system; and said cluster file system reading one or more of said plurality of shard objects of said given file from said second storage tier using said shard object identifiers, wherein said translation shim transforms object get operations from said second storage tier to file block read input/output operations.
14. The method of claim 11 , wherein, in response to said client of said cluster file system opening said given file with a predefined attribute, said translation shim selects a substantially fastest tier that stores said given file when said given file is stored on a plurality of storage tiers and obtains said layout for said given file and storage address information for said given file from a said bucket on said second storage tier corresponding to said given file.
15. The method of claim 11 , wherein said translation shim is further configured to detect a request by said client of said cluster file system to archive said given file, and provide a data movement command to a copy agent of a Hierarchical Storage Manager on said cluster file system to archive said given file on said second storage tier.
16. The method of claim 11 , further comprising the steps of intercepting a file lookup request for a file from a file system client and providing said client with location information for copies of said file on one or more of said first storage tier and said second storage tier, and wherein said file system client performs an input/output operation on said file using said location information.
A method for managing file storage across multiple storage tiers in a distributed file system addresses the challenge of efficiently accessing files stored in different storage tiers with varying performance characteristics. The method involves intercepting a file lookup request from a file system client for a specific file and providing the client with location information for copies of the file stored across one or more storage tiers, such as a high-performance first tier and a lower-cost second tier. The file system client then uses this location information to perform input/output operations on the file, ensuring optimal access based on the file's availability and performance requirements. This approach enhances file retrieval efficiency by dynamically directing clients to the most appropriate storage location, reducing latency and improving overall system performance. The method supports seamless integration with existing file system architectures, allowing clients to interact with files transparently while leveraging the benefits of tiered storage. By dynamically providing location information, the system ensures that clients can access files from the most suitable storage tier without manual intervention, optimizing resource utilization and performance.
17. A computer program product, comprising a non-transitory machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps: communicating over a network with (i) a cluster file system on a first storage tier, and (ii) a second storage tier, wherein the first and second storage tiers comprise respective disjoint subsets of storage, wherein said first storage tier has a substantially lower access time than the second storage tier; providing a client, on said compute node, of said cluster file system with access to one or more files in said cluster file system on said first storage tier using a first object protocol; and executing, by said compute node, a translation shim that converts between the first object protocol and a different, second object protocol corresponding to said second storage tier to provide said client of said cluster file system with one or more of direct read access and direct write access to one or more files on said second storage tier using the first object protocol, wherein said translation shim provides said client of said cluster file system with direct write access to a given file on said second storage tier by at least storing, in a bucket on said second storage tier, (i) a layout for said given file, wherein the layout comprises at least one parameter indicating a location of said given file within said cluster file system on said first storage tier, and (ii) a plurality of shard objects of said given file.
18. The computer program product of claim 17 , wherein said layout for said given file is obtained from a metadata warehouse corresponding to said cluster file system on said first storage tier, and wherein said layout is stored as an object in said bucket on said second storage tier.
This invention relates to a system for managing file layouts in a distributed storage environment with multiple storage tiers. The problem addressed is efficiently accessing and storing file layout information across different storage tiers in a cluster file system. The solution involves a computer program product that retrieves a file's layout from a metadata warehouse associated with the cluster file system on a first storage tier, typically high-performance storage. The layout, which defines the file's structure and location, is then stored as an object in a bucket on a second storage tier, such as a lower-cost, higher-capacity storage system. This approach ensures that layout information remains accessible while optimizing storage costs and performance. The system dynamically manages file layouts by retrieving them from the metadata warehouse when needed and persisting them in the bucket for future access. This method supports efficient file operations by maintaining up-to-date layout information across storage tiers, improving data retrieval and management in distributed storage environments. The invention is particularly useful in large-scale storage systems where balancing performance and cost is critical.
19. The computer program product of claim 17 , wherein said direct read access of said given file from said second storage tier comprises said translation shim obtaining storage location information from a metadata warehouse, obtaining contents of said bucket on said second storage tier storing said layout for said given file and said plurality of shard objects of said given file and providing shard object identifiers on said second storage tier to said client of said cluster file system; and said cluster file system reading one or more of said plurality of shard objects of said given file from said second storage tier using said shard object identifiers, wherein said translation shim transforms object get operations from said second storage tier to file block read input/output operations.
20. The computer program product of claim 17 , wherein, in response to said client of said cluster file system opening said given file with a predefined attribute, said translation shim selects a substantially fastest tier that stores said given file when said given file is stored on a plurality of storage tiers and obtains said layout for said given file and storage address information for said given file from said bucket on said second storage tier corresponding to said given file.
Unknown
January 26, 2021
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.