Near-Duplicate Filtering in Search Engine Result Page of an Online Shopping System

1. A computer-implemented method to reduce same merchant near-duplicate entries in online shopping system search results, comprising: for each pair of entries in a set of entries from the same merchant, each entry characterizing a product in a data store of an online shopping system and each entry characterized by a set of quantified attributes, determining, by one or more computing devices, a distance between the entries in the pair in a vector space of the quantified attributes; determining, by the one or more computing devices, clusters of entries as a function of the determined distance between each pair of entries; receiving, by the one or more computing devices, a query directed to the data store; and returning, by the one or more computing devices, an ordered list of results responsive to the query from the data store of an online shopping system, filtered by at least one of the following: excluding from the ordered list all but a predetermined first threshold number of entries from each cluster of entries; and after including in the ordered list a first entry, excluding entries within a predetermined threshold distance of the first entry.

2. The method of claim 1 , wherein determining a distance between the entries in the pair based on the quantified attributes comprises determining a weighted sum of an edit distance between the quantified attributes of entries of the pair.

3. The method of claim 2 , wherein the edit distance is one of: a Hamming distance, a Levenshtein distance, a Damerau-Levenshtein distance, and a Jaro-Winkler distance.

4. The method of claim 1 , wherein determining clusters of entries comprises identifying as clusters groups of entries related to another entry by a determined distance of less than a predetermined first threshold distance.

5. The method of claim 4 , wherein the distance is normalized on an interval from 0 to 1, and the predetermined threshold distance is approximately 0.05.

6. The method of claim 1 , wherein filtering by at least one of the following comprises: excluding entries characterized by a minimum distance to nearest entry less than a predetermined third threshold; and including only entries from a given cluster of entries.

7. The method of claim 1 , further comprising: for at least one result in the ordered list, returning, by the one or more computing devices, a link: which, when selected, prompts as a response from the online shopping system an ordered list of products in the same cluster as the at least one result.

8. A computer program product, comprising: a non-transitory computer-readable storage device having computer-executable program instructions embodied thereon that when executed by a computer cause the computer to reduce same merchant near-duplicate entries in online shopping system search results, the computer-executable program instructions comprising: computer-executable program instructions to determine, for each pair of entries from the same merchant in a set of entries, each entry characterizing a product in a data store of an online shopping system and each entry characterized by a set of quantified attributes, a distance between the entries in the pair in a vector space of the quantified attributes; computer-executable program instructions to determine clusters of entries from a graph formed with each determined distance as an edge between nodes representing the entries used to determine the corresponding distance, each entry cluster identified by a cluster identifier; computer-executable program instructions to receive a query directed to the data store; and computer-executable program instructions to return an ordered list of results responsive to the query from the data store of an online shopping system, filtered by at least one of the following: excluding from the ordered list all but a predetermined first threshold number of entries from each cluster of entries; and after including in the ordered list a first entry, excluding entries within a predetermined threshold distance of the first entry.

9. The computer program product of claim 8 , wherein determining a distance between the entries in the pair based on the quantified attributes comprises determining a weighted sum of an edit distance between the quantified attributes of entries of the pair.

10. The computer program product of claim 9 , wherein the edit distance is one of: a Hamming distance, a Levenshtein distance, a Damerau-Levenshtein distance, and a Jaro-Winkler distance.

11. The computer program product of claim 8 , wherein determining clusters of entries comprises identifying as clusters groups of entries related to another entry by a determined distance of less than a predetermined first threshold distance.

12. The computer program product of claim 11 , wherein the distance is normalized on an interval from 0 to 1, and the predetermined threshold distance is approximately 0.05.

13. The computer program product of claim 8 , wherein filtering by at least one of the following comprises: excluding entries characterized by a minimum distance to nearest entry less than a predetermined third threshold; and including only entries from a given cluster.

14. The computer program product of claim 8 , further comprising: for at least one result in the ordered list, returning, by the one or more computing devices, a link: which, when selected, prompts as a response from the online shopping system an ordered list of products in the same cluster as the at least one result.

15. A system to reduce same merchant near-duplicate entries in online shopping system search results, comprising: a storage device; and a processor communicatively coupled to the storage device, wherein the processor executes application code instructions that are stored in the storage device to cause the system to: determine, for each pair of entries in a set of entries from the same merchant, each entry characterizing a product in a data store of an online shopping system and each entry characterized by a set of quantified attributes, a distance between the entries in the pair in a vector space of the quantified attributes; determine clusters of entries from a graph formed with each determined distance as an edge between nodes representing the entries used to determine the corresponding distance, each entry cluster identified by a cluster identifier; receive a query directed to the data store; and return an ordered list of results responsive to the query from the data store of an online shopping system, filtered by at least one of the following: excluding from the ordered list all but a predetermined first threshold number of entries from each cluster of entries; and after including in the ordered list a first entry, excluding entries within a predetermined threshold distance of the first entry.

16. The computer program product of claim 15 , wherein determining a distance between the entries in the pair based on the quantified attributes comprises determining a weighted sum of an edit distance between the quantified attributes of entries of the pair.

17. The system of claim 16 , wherein the edit distance is one of: a Hamming distance, a Levenshtein distance, a Damerau-Levenshtein distance, and a Jaro-Winkler distance.

18. The system of claim 15 , wherein determining clusters of entries comprises identifying as clusters groups of entries related to another entry by a determined distance of less than a predetermined first threshold distance.

19. The system of claim 18 , wherein the distance is normalized on an interval from 0 to 1, and the predetermined threshold distance is approximately 0.05.

20. The system of claim 15 , wherein filtering by at least one of the following comprises: excluding entries characterized by a minimum distance to nearest entry less than a predetermined third threshold; and including only entries from a given cluster.

21. The system of claim 15 , further comprising: for at least one result in the ordered list, returning, by one or more computing devices, a link which, when selected, prompts as a response from the online shopping system an ordered list of products in the same cluster as the at least one result.