Ranking Content Items Based on Preference Scores

1. A method for re-ordering a plurality of content items using ranking preference data and user-interaction data comprising: receiving identifiers of a plurality of content items by a computing device, wherein each content item has a position in a first order of the plurality of content items; determining content items from the plurality of content items that are relevant content items by the computing device; receiving ranking preference data comprising a first parameter and a second parameter by the computing device, wherein the first parameter indicates a preference of a user for content items that are relevant content items and wherein the second parameter indicates a preference of the user to maintain the first order such that for any two content items that have been determined to be in the same relevant or irrelevant class based on the user-interaction data, the priority for those content items remains the same in a second order as in the first order, wherein relative weights of the first parameter and the second parameter are selectable by the user, and wherein the ranking preference data further comprises a value that represents a preference to rank the content items based on interaction data associated with a different user; calculating a gain value for each content item based on a sum of preference scores for that content item with respect to each of the other content items of the plurality of content items; and generating the second order for the plurality of content items using the ranking preference data and the gain value by the computing device.

2. The method of claim 1 , further comprising receiving user-interaction data associated with the user and one or more of the plurality of content items by the computing device.

3. The method of claim 2 , wherein determining content items from the plurality of content items that are relevant content items comprising determining content items from the plurality of content items that are identified by the user-interaction data.

4. The method of claim 2 , wherein the user-interaction data is received from a search engine.

5. The method of claim 2 , wherein the user-interaction data comprises one or more of dwell-time and click data.

6. The method of claim 1 , wherein the second order is different than the first order.

7. The method of claim 1 , further comprising training a ranking model using the second order.

8. The method of claim 1 , wherein generating the second order for the plurality of content items using the ranking preference data comprises: for each content item: calculating a preference score for the content item with respect to each of the other content items in the first order based on the first parameter, the second parameter, and whether or not the content item was determined to be a relevant content item; and calculating the gain value for the content item based on a sum of the calculated preference scores for the content item; and generating the second order using the gain value calculated for each content item.

9. A method for re-ordering a plurality of content items using ranking preference data and user-interaction data comprising: receiving identifiers of a plurality of content items by a computing device, wherein each content item has a position in a first order of the plurality of content items; receiving user-interaction data associated with one or more users and one or more of the plurality of content items by the computing device; determining content items from the plurality of content items that are relevant content items based on the user-interaction data by the computing device; determining content items from the plurality of content items are irrelevant content items based on the user-interaction data by the computing device; receiving ranking preference data associated with a user by the computing device, wherein the ranking preference data comprises a first parameter and a second parameter, wherein the first parameter indicates a preference of the one or more users for content items that are relevant content items and wherein the second parameter indicates a preference of the user to maintain the first order such that for any two content items that have been determined to be in the same relevant or irrelevant class based on the user-interaction data, the priority for those content items remains the same in a second order as in the first order, wherein relative weights of the first parameter and the second parameter are selectable by the user, and wherein the ranking preference data further comprises a value that represents a preference to rank the content items based on interaction data associated with a different user; calculating a gain value for each content item based on a sum of preference scores for that content item with respect to each of the other content items of the plurality of content items; and generating the second order for the plurality of content items by the computing device based on the determined content items that are relevant content items, the determined content items that are irrelevant content items, the ranking preference data, and the gain value.

10. The method of claim 9 , wherein the content items comprise one or more of web pages, images, videos, or documents.

11. The method of claim 9 , further comprising training a ranking model using the second order.

12. The method of claim 9 , wherein the second order is different than the first order.

13. The method of claim 9 , wherein the user-interaction data comprises one or more of dwell-time and click data.

14. The method of claim 9 , wherein the ranking preference data is specified by the one or more users.

15. The method of claim 9 , wherein generating the second order for the plurality of content items based on the determined content items that are relevant content items, the determined content items that are irrelevant content items, and the ranking preference data comprises: for each content item: calculating a preference score for the content item with respect to each of the other content items based on the first parameter, the second parameter; and whether the content item was determined to be a relevant content item or an irrelevant content item; calculating the gain value for the content item based on a sum of the calculated preference scores for the content item; and generating the second order using the gain calculated for each content item.

16. A system for re-ordering a plurality of web pages using ranking preference data and user-interaction data comprising: at least one computing device configured to generate a query for a user; and a user-interaction ranker configured to: receive identifiers of a plurality of web pages, wherein each web page has a position in a first order of the plurality of web pages and each web page is responsive to the generated query; determine web pages from the plurality of web pages that are relevant web pages; receive ranking preference data comprising a first parameter and a second parameter, wherein the first parameter indicates a preference of the user for web pages that are relevant web pages and wherein the second parameter indicates a preference of the user to maintain the first order such that for any two web pages that have been determined to be in the same relevant or irrelevant class based on the user-interaction data, the priority for those web pages remains the same in a second order as in the first order, wherein relative weights of the first parameter and the second parameter are selectable by the user, and wherein the ranking preference data further comprises a value that represents a preference to rank the web pages based on interaction data associated with a different user; calculate a gain value for each web page based on a sum of preference scores for that web page with respect to each of the other web pages of the plurality of web pages; and generate the second order for the plurality of web pages using the ranking preference data and the gain value.

17. The system of claim 16 , wherein the user-interaction ranker is further configured to receive user-interaction data associated with the user and one or more of the plurality of web pages.

18. The system of claim 16 , wherein the user-interaction ranker configured to determine web pages from the plurality of web pages that are relevant web pages comprises the user-interaction ranker configured to determine web pages from the plurality of web pages that are identified by the user-interaction data.

19. The system of claim 16 , wherein the user-interaction ranker configured to generate the second order for the plurality of web pages using the ranking preference data comprises the user-interaction ranker configured to: for each web page: calculate a preference score for the web page with respect to each of the other web pages in the first order based on the first parameter, the second parameter, and whether or not the web page was determined to be a relevant web page; and calculate the gain value for the web page based on a sum of the calculated preference scores for the web page; and generate the second order using the gain value calculated for each web page.