Prefetch Mechanism for Tiered Streaming Storage Systems

1. A streaming storage system, comprising: a segment store to process data requests associated with a data stream, wherein the segment store includes: first tier storage for storing a most recent portion of ingested data; a cache for storing data served to a client; a segment container to: execute storage operations for segments allocated to the segment container; maintain information indicative of which portions of ingested data are stored in the cache; initiate retrieval of data from second tier storage when a data request addresses a segment not currently stored in the first tier storage or the cache; monitor and evaluate data request information, wherein evaluation of the data request information includes a determination of whether monitored data requests associated with a particular reader constitute sequential data requests based, at least in part, on stream offsets for successive data requests being less than a predefined prefetch length; and responsive to identifying a prefetch trigger based on the data request information, initiate a prefetch read of data stored in the second tier storage.

2. The streaming storage system of claim 1, wherein evaluation of the data request information includes a determination of whether monitored data requests associated with a particular reader constitute sequential data requests.

3. The streaming storage system of claim 2, wherein monitored data requests constitute sequential data requests only if successive data requests target monotonically increasing stream offsets.

4. The streaming storage system of claim 2, wherein no prefetch trigger occurs when the evaluation determines that monitored data requests are associated with a non-sequential reader.

5. The streaming storage system of claim 1, wherein initiating a prefetch comprises initiating a prefetch asynchronously with respect to processing of data requests.

6. The streaming storage system of claim 1, wherein the data request information includes a segment identifier and a reader identifier indicative of a reader associated with a data request.

7. The streaming storage system of claim 1, further comprising a streaming storage controller to process metadata requests associated with the data stream.

8. The streaming storage system of claim 1, wherein a storage latency associated with the first tier storage is lower than a storage latency associated with the second tier storage.

9. The streaming storage system of claim 1, wherein the first tier storage include Apache Bookkeeper storage and the second tier storage comprises Apache Hadoop Distributed File System (HDFS) storage.

10. A streaming storage method, comprising: processing, by a segment store of a streaming storage system, data requests associated with a data stream, wherein the segment store includes: first tier storage for storing a most recent portion of ingested data; a cache for storing data served to a client; a segment container to: execute storage operations for segments allocated to the segment container; maintain information indicative of which portions of ingested data are stored in the cache; initiate retrieval of data from second tier storage when a data request addresses a segment not currently stored in the first tier storage or the cache; monitor and evaluate data request information, wherein evaluation of the data request information includes a determination of whether monitored data requests associated with a particular reader constitute sequential data requests based, at least in part, on stream offsets for successive data requests being less than a predefined prefetch length; and responsive to identifying a prefetch trigger based on the data request information, initiate a prefetch read of data stored in the second tier storage.

11. The method of claim 10, wherein evaluation of the data request information includes a determination of whether monitored data requests associated with a particular reader constitute sequential data requests.

12. The method of claim 11, wherein monitored data requests constitute sequential data requests only if successive data requests target monotonically increasing stream offsets.

13. The method of claim 11, wherein no prefetch trigger occurs when the evaluation determines that monitored data requests are associated with a non-sequential reader.

14. The method of claim 10, wherein initiating a prefetch comprises initiating a prefetch asynchronously with respect to processing of data requests.

15. The method of claim 10, wherein the data request information includes a segment identifier and a reader identifier indicative of a reader associated with a data request.

16. The method of claim 10, further comprising a streaming storage controller to process metadata requests associated with the data stream.

17. The method of claim 10, wherein a storage latency associated with the first tier storage is lower than a storage latency associated with the second tier storage.

18. The method of claim 10, wherein the first tier storage comprises Apache Bookkeeper storage and the second tier storage comprises Apache Hadoop Distributed File System (HDFS) storage.