System and Method for Improving Data Compression of a Storage System in an Online Manner

1. A computer-implemented method for improving data compression of data chunks of a storage system, the method comprising: in response to a sequence of data to be stored, partitioning the sequence of data into a plurality of data chunks according to a predetermined chunking algorithm; generating, by using a computer system, a sketch for each data chunk of the data chunks based on one or more features extracted from the data chunk; performing a lookup operation in a sketch index based on the sketch of the each data chunk to determine a compression region identifier (ID) of an existing compression region, wherein the sketch index maps a particular sketch to a particular compression region; associating said each data chunk of the data chunks of the sequence of data with one group of a plurality of groups based on the particular sketch, wherein each group of the plurality of groups is represented by a sketch, wherein the associating each data chunk of the data chunks of the sequence of data comprises merging the data chunk of the sequence of data with data chunks of an existing compression region identified by a corresponding compression region ID, wherein the merging comprises reorganizing the data chunks of the sequence of data and the data chunks of the existing compression region from an original sequence order to a second sequence order, similar data chunks being positioned adjacent to each other; and compressing and storing data chunks of the each group in a corresponding existing compression region of the storage system so that the similar data chunks are compressed and stored in the same compression region, wherein the compressing and storing data chunks of each group comprises writing data chunks of each group of the groups whose data chunks reach a predetermined threshold to a respective compression region and reclaiming a previous compression region space.

2. The method of claim 1 , wherein each group is associated with a buffer that stores a plurality of data chunks from the sequence of data that have the same sketch.

3. The method of claim 2 , wherein compressing and storing data chunks of each group comprises: determining whether a number of data chunks of each group of the groups reaches a predetermined threshold.

4. The method of claim 1 , wherein each group is associated with a buffer that buffers a plurality of existing data chunks having the same sketch that have been previously stored in the storage system.

5. The method of claim 4 , wherein associating each of the data chunk of the sequence of data with one of a plurality of groups based on the sketch comprises: identifying an existing compression region based on the sketch of data chunk of the sequence of data; and merging the data chunk of the sequence of data with data chunks of the identified existing compression region.

6. The method of claim 5 , further comprising: retrieving and decompressing data chunks from the identified existing compression region; and storing the data chunks in one of the groups associated with the sketch of the data chunk.

7. The method of claim 6 , further comprising writing the group of the merged data chunks to a new compression region.

8. The method of claim 7 , wherein the previous compression region space is reclaimed after the group of the merged data chunks have been written to the new compression region.

9. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for improving data compression of data chunks of a storage system, the operations comprising: in response to a sequence of data to be stored, partitioning the sequence of data into a plurality of data chunks according to a predetermined chunking algorithm; generating a sketch for each data chunk of the data chunks based on one or more features extracted from the data chunk; performing a lookup operation in a sketch index based on the sketch of the each data chunk to determine a compression region identifier (ID) of an existing compression region, wherein the sketch index maps a particular sketch to a particular compression region; associating said each data chunk of the data chunks of the sequence of data with one of a plurality of groups based on the particular sketch, wherein each group of the plurality of groups is represented by a sketch, wherein the associating each data chunk of the data chunks of the sequence of data comprises merging the data chunk of the sequence of data with data chunks of an existing compression region identified by a corresponding compression region ID, wherein the merging comprises reorganizing the data chunks of the sequence of data and the data chunks of the existing compression region from an original sequence order to a second sequence order, similar data chunks being positioned adjacent to each other; and compressing and storing data chunks of the each group in a corresponding existing compression region of the storage system so that the similar data chunks are compressed and stored in the same compression region, wherein the compressing and storing data chunks of each group comprises writing data chunks of each group of the groups whose data chunks reach a predetermined threshold to a respective compression region and reclaiming a previous compression region space.

10. The non-transitory machine-readable medium of claim 9 , wherein each group is associated with a buffer that stores a plurality of data chunks from the sequence of data that have the same sketch.

11. The non-transitory machine-readable medium of claim 10 , wherein compressing and storing data chunks of each group comprises: determining whether a number of data chunks of each group of the groups reaches a predetermined threshold.

12. The non-transitory machine-readable medium of claim 9 , wherein each group is associated with a buffer that buffers a plurality of existing data chunks having the same sketch that have been previously stored in the storage system.

13. The non-transitory machine-readable medium of claim 12 , wherein associating each data chunk of the data chunks of the sequence of data with one of a plurality of groups based on the sketch comprises: identifying an existing compression region based on the sketch of data chunk of the sequence of data; and merging the data chunk of the sequence of data with data chunks of the identified existing compression region.

14. The non-transitory machine-readable medium of claim 13 , wherein the operations further comprise: retrieving and decompressing data chunks from the identified existing compression region; and storing the data chunks in one of the groups associated with the sketch of the data chunk.

15. The non-transitory machine-readable medium of claim 14 , wherein the operations further comprise writing the group of the merged data chunks to a new compression region.

16. The non-transitory machine-readable medium of claim 15 , wherein the previous compression region space is reclaimed after the group of the merged data chunks have been written to the new compression region.

17. A data processing system, comprising: a processor; and a memory coupled to the processor for storing instructions, which when executed by the processor, cause the processor to perform operations, the operations including: in response to a sequence of data to be stored, partitioning the sequence of data into a plurality of data chunks according to a predetermined chunking algorithm, generating a sketch for each data chunk of the data chunks based on one or more features extracted from the data chunk, performing a lookup operation in a sketch index based on the sketch of the data chunk to determine a compression region identifier (ID) of an existing compression region, wherein the sketch index maps a particular sketch to a particular compression region, associating said each data chunk of the data chunks of the sequence of data with one of a plurality of groups based on the particular ketch, wherein each group is represented by a sketch, wherein the associating each data chunk of the data chunks of the sequence of data comprises merging the data chunk of the sequence of data with data chunks of an existing compression region identified by a corresponding compression region ID, wherein the merging comprises reorganizing the data chunks of the sequence of data and the data chunks of the existing compression region from an original sequence order to a second sequence order, similar data chunks being positioned adjacent to each other, and compressing and storing data chunks of each group in a corresponding existing compression region of the storage system so that the similar data chunks are compressed and stored in the same compression region, wherein the compressing and storing data chunks of each group comprises writing data chunks of each group of the groups whose data chunks reach a predetermined threshold to a respective compression region and reclaiming a previous compression region space.

18. The system of claim 17 , wherein each group is associated with a buffer that stores a plurality of data chunks from the sequence of data that have the same sketch.

19. The system of claim 18 , wherein compressing and storing data chunks of each group comprises: determining whether a number of data chunks of each group of the groups reaches a predetermined threshold.

20. The system of claim 17 , wherein each group is associated with a buffer that buffers a plurality of existing data chunks having the same sketch that have been previously stored in the storage system.

21. The system of claim 20 , wherein associating each data chunk of the data chunks of the sequence of data with one of a plurality of groups based on the sketch comprises: identifying an existing compression region based on the sketch of data chunk of the sequence of data; and merging the data chunk of the sequence of data with data chunks of the identified existing compression region.

22. The system of claim 21 , wherein the operations further comprise: retrieving and decompressing data chunks from the identified existing compression region; and storing the data chunks in one of the groups associated with the sketch of the data chunk.

23. The system of claim 22 , wherein the operations further comprise writing the group of the merged data chunks to a new compression region.

24. The system of claim 23 , wherein the previous compression region space is reclaimed after the group of the merged data chunks have been written to the new compression region.