A data storage service stores a dataset on a set of storage nodes in accordance with a first encoding. A set of shards constituting quorum, and one or more additional shards, are stored on the storage nodes. The data storage system determines to store the dataset according to a second encoding, in which the second encoding has fewer total shards. The data storage system reconfigures the storage of the dataset in accordance with the second encoding, such that the reconfigured storage comprises subsets of shards from the first encoding that were not re-encoded in forming the second encoding.
Legal claims defining the scope of protection, as filed with the USPTO.
1. A system, comprising: a plurality of storage nodes; and at least one processor; a memory having stored thereon processor-executable instructions that, in response to being executed by the at least one processor, cause the system to at least: store a dataset on the plurality of storage nodes according to a first encoding in which the dataset is stored as n shards, wherein the n shards comprise at least a quorum of k shards stored on k of the plurality of storage nodes; determine to store the dataset according to a second encoding, wherein the dataset is stored according to the second encoding using fewer than n shards and quorum comprises less than k shards; and reconfigure the plurality of storage nodes to store the dataset according to the second encoding, wherein at least one of the storage nodes is dropped by the reconfiguration, and wherein a subset of a shard stored on the dropped node is copied to another node without re-encoding.
2. The system of claim 1 , wherein the second encoding is selected based at least in part on storage efficiency criteria.
3. The system of claim 1 , wherein the second encoding is selected based at least in part on resiliency criteria.
4. The system of claim 1 , wherein the reconfigured plurality of storage nodes comprises less than n storage nodes.
5. The system of claim 1 , wherein the k shards are identity shards.
6. A method, comprising: storing a dataset according to a first encoding in which the dataset is stored as a plurality of shards on a plurality of storage nodes; selecting a second encoding for storing the dataset, the second encoding comprising fewer shards than the first encoding; and reconfiguring the plurality of storage nodes to store the dataset according to the second encoding, wherein one of a plurality of portions of a shard of the first encoding is combined, without re-encoding, with a shard of the second encoding.
7. The method of claim 6 , further comprising: splitting the shard of the first encoding into the plurality of portions; combining the plurality of portions with shards of the second encoding; and dropping the shard of the first encoding.
8. The method of claim 6 , wherein the second encoding is selected based at least in part on storage efficiency criteria.
9. The method of claim 6 , wherein the second encoding is selected based at least in part on a reduction in a requirement for resiliency to data loss.
10. The method of claim 6 , wherein the shard of the second encoding comprises at least one portion that was not copied from another node by the reconfiguring.
11. The method of claim 6 , wherein the second encoding comprises re-encoded data, the re-encoded data based on access to less than all of the plurality of storage nodes.
12. The method of claim 6 , wherein the shard of the second encoding is generated based on access to less than all of the plurality of storage nodes.
13. The method of claim 6 , further comprising: selecting the second encoding based at least in part on a ratio of shards needed for quorum in the first encoding to shards needed for quorum in the second encoding.
14. A non-transitory computer-readable storage medium having stored thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least: store a dataset on a plurality of storage nodes according to a first encoding comprising a plurality of shards stored on a plurality of storage nodes; select a second encoding for the dataset, based at least in part on the second encoding comprising fewer shards than the first encoding; and store the dataset on the plurality of storage nodes according to the second encoding, wherein a shard of the first encoding is dropped and a portion of the dropped shard is combined with another shard without re-encoding.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the second encoding is selected in response to a request to alter at least one of a performance or resiliency characteristic.
16. The non-transitory computer-readable storage medium of claim 14 , wherein the second encoding is selected in response to a change to at least one of storage efficiency criteria or resiliency criteria.
17. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to reconfigure operation of the plurality of storage nodes to store the dataset according to the second encoding.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to select one or more of the plurality of storage nodes to exclude from the reconfigured plurality of storage nodes.
19. The non-transitory computer-readable storage medium of claim 17 , wherein reconfiguring operation of the plurality of storage nodes comprises re-routing data retrieval operations in accordance with the second encoding.
20. The non-transitory computer-readable storage medium of claim 14 , wherein a shard of the second encoding is re-encoded based on a subset of the plurality of shards.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
June 27, 2018
September 15, 2020
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.