Frequent Items Detection in Data Sets

1. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor of a database management system that includes a memory, cause the database management system to estimate frequencies of a predetermined count of most common items in a data set in a single pass through the data set by performing a method, the method comprising: allocating a first portion of the memory for an unprotected list of a first predetermined size for storing a first count of keys representing items of the data set sorted by estimated frequency; creating the unprotected list in the first portion of the memory; allocating a second portion of the memory for a protected list of a second predetermined size for storing a second count of keys representing items of the data set sorted by hit count; creating the protected list in the second portion of the memory; receiving a new item from the data set; determining whether a key corresponding to the new item is stored in the unprotected list or the protected list; updating the estimated frequency of the new item when it is determined that that the key corresponding to the new item is stored in the unprotected list; updating the hit count of the new item when it is determined that the key corresponding to the new item is stored in the protected list; storing the key corresponding to the new item in the unprotected list when the key satisfies criteria for storage in the unprotected list; evicting a key corresponding to an item having a lowest estimated frequency from the unprotected list in response to the storing when the unprotected list is full; transferring at least one item having a hit count that exceeds a threshold hit count from the unprotected list to the protected list; and transferring an item having the lowest hit count from the protected list to the unprotected list when the protected list is full.

2. The non-transitory computer readable medium of claim 1 , wherein the item transferred from the protected list to the unprotected list corresponds to a minimum hit count stored in the protected list, a position of the item transferred from the protected list to the unprotected list is based on the item's estimated frequency, a sum of the first count and the second count is equal to the predetermined count, and the threshold hit count is equal to a lowest hit count of any item stored in the protected list.

3. The non-transitory computer readable medium of claim 1 , wherein the item transferred from the protected list to the unprotected list corresponds to a minimum hit count stored in the protected list, wherein the data set is stored by the database management system.

4. The non-transitory computer readable medium of claim 3 , wherein each item group from the one or more item groups comprises a group count which is incremented at every occurrence of an item in the data set belonging to that item group.

5. The non-transitory computer readable medium of claim 4 , wherein the unprotected list is comprised of distinct items whose occurrence in the data set causes the group count to exceed a minimum estimated frequency stored in the unprotected list.

6. The non-transitory computer readable medium of claim 5 , wherein the estimated frequency for each item in the unprotected list has an initial value and a maximum error value, and wherein the maximum error value associated with the estimated frequency is one less than the initial value associated with the estimated frequency.

7. The non-transitory computer readable medium of claim 6 , wherein the protected list comprises a hash map for storing the estimated frequency and the maximum error for each item in the protected list, and an array data structure for holding the hit count for each item in the protected list.

8. The non-transitory computer readable medium of claim 1 , wherein the unprotected list comprises a double-linked Stream Summary data structure for storing item key and estimated frequency information and a hash map for holding pointers into the double-linked Stream Summary data structure.

9. The non-transitory computer readable medium of claim 1 , wherein the unprotected list and the protected list are represented by a single two dimensional array structure, wherein the items associated with the unprotected list are stored together at a front of the two dimensional array and sorted by the estimated frequency, and wherein the items associated with the protected list are stored together at a back of the two dimensional array and sorted by the hit count.

10. A computer-implemented method for generating a list of most common items and respective item frequencies from a data set comprising a plurality of items in database management system that includes a memory, the computer-implemented method comprising: allocating a first portion of the memory for an unprotected list of a first predetermined size for storing a first count of keys representing items of the data set sorted by estimated frequency; creating the unprotected list in the first portion of the memory; allocating a second portion of the memory for a protected list of a second predetermined size for storing a second count of keys representing items of the data set sorted by hit count; creating the protected list in the second portion of the memory; receiving a new item from the data set; determining whether a key corresponding to the new item is stored in the unprotected list or the protected list; updating the estimated frequency of the new item when it is determined that that the key corresponding to the new item is stored in the unprotected list; updating the hit count of the new item when it is determined that the key corresponding to the new item is stored in the protected list; storing the key corresponding to the new item in the unprotected list when the key satisfies criteria for storage in the unprotected list; evicting a key corresponding to an item having a lowest estimated frequency from the unprotected list in response to the storing when the unprotected list is full; transferring at least one item having a hit count that exceeds a threshold hit count from the unprotected list to the protected list; and transferring an item having the lowest hit count from the protected list to the unprotected list when the protected list is full.

11. The computer-implemented method of claim 10 , wherein the item transferred from the protected list to the unprotected list corresponds to a minimum hit count stored in the protected list, and a position of the item transferred from the protected list to the unprotected list is based on the item's estimated frequency.

12. The computer-implemented method of claim 10 , wherein the transferred from the protected list to the unprotected list corresponds to a minimum hit count stored in the protected list; and wherein each item group from the one or more item groups comprises a group count which is incremented at every occurrence of an item in the data set belonging to that item group.

13. The computer-implemented method of claim 12 , wherein the unprotected list is comprised of distinct items whose occurrence in the data set causes the group count to exceed a minimum estimated frequency stored in the unprotected list.

14. The computer-implemented method of claim 13 , wherein the estimated frequency for each item in the unprotected list has an initial value and a maximum error value; and wherein the maximum error value associated with the estimated frequency is one less than the initial value associated with the estimated frequency.

15. The computer-implemented method of claim 14 , wherein the protected list comprises a hash map for storing the estimated frequency and the maximum error for each item in the protected list, and an array data structure for holding the hit count for each item in the protected list.

16. The computer-implemented method of claim 10 , wherein the unprotected list of items and the protected list are represented by a single two dimensional array structure, wherein the items associated with the unprotected list are stored together at a front of the two dimensional array and sorted by the estimated frequency; and wherein the items associated with the protected list are stored together at a back of the two dimensional array and sorted by the hit count.

17. A system for generating a list of most common items and respective item frequencies from a data set comprising a plurality of items, comprising one or more processors operatively connected to a data storage and a memory and configured as: a data receiving module configured to receive the data set comprising the plurality of items; and a data item frequency estimation module configured to: allocate a first portion of the memory for an unprotected list of a first predetermined size for storing a first count of keys representing items of the data set sorted by estimated frequency; create the unprotected list in the first portion of the memory; allocate a second portion of the memory for a protected list of a second predetermined size for storing a second count of keys representing items of the data set sorted by hit count; create the protected list in the second portion of the memory; receive a new item from the data set; determine whether a key corresponding to the new item is stored in the unprotected list or the protected list; update the estimated frequency of the new item when it is determined that that the key corresponding to the new item is stored in the unprotected list; update the hit count of the new item when it is determined that the key corresponding to the new item is stored in the protected list; store the key corresponding to the new item in the unprotected list when the key satisfies criteria for storage in the unprotected list; evict a key corresponding to an item having a lowest estimated frequency from the unprotected list in response to the storing when the unprotected list is full; transfer at least one item having a hit count that exceeds a threshold hit count from the unprotected list to the protected list; and transfer an item having the lowest hit count from the protected list to the unprotected list when the protected list is full.

18. The system of claim 17 , wherein the item transferred from the protected list to the unprotected list corresponds to a minimum hit count stored in the protected list, a position of the item transferred from the protected list to the unprotected list is based on the item's estimated frequency, and the threshold hit count is equal to a lowest hit count of any item stored in the protected list.

19. The system of claim 17 , wherein the protected list comprises a hash map for storing the estimated frequency and the maximum error for each item in the protected list, and an array data structure for holding the hit count for each item in the protected list.

20. The system of claim 17 , wherein the unprotected list and the protected list are represented by a single two dimensional array structure; wherein the items associated with the unprotected list are stored together at a front of the two dimensional array and sorted by the estimated frequency; and wherein the items associated with the protected list are stored together at a back of the two dimensional array and sorted by the hit count.