Systems and methods for generating and maintaining internet user profile data

1. A system, comprising: a user profile database storing a plurality of user profiles and at least one cookie set associated with each user profile; and at least one processor configured to: store a cookie set of a selected one of the user profiles in cookie storage of a web crawler; load a publisher web page at a publisher server while allowing the publisher server to access the cookie set; identify a user-specific advertisement that has been placed on the publisher web page, based on the publisher server access to the cookie set, by an advertisement channel that is separate from the at least one processor; and identify a placement pathway by which the identified user-specific advertisement was placed on the publisher web page by the advertisement channel.

2. The system of claim 1 , wherein the at least one processor is configured to identify the placement pathway by identifying a site-specific data tree for the identified user-specific advertisement, and wherein the site-specific data tree comprises a base uniform resource locator (URL) associated with a corresponding advertisement creative and branches indicating the placement pathway of the corresponding advertisement creative.

3. The system defined in claim 1 , further comprising: a database storing site-specific segmentation data that is specific to each of a plurality of publisher websites, wherein the site-specific segmentation data for each publisher website includes user population statistics related to a population of human users of that website.

4. The system of claim 3 , wherein the at least one processor is further configured to generate the at least one cookie set associated with each user profile at least in part by selecting a plurality of publisher websites to crawl based on the site-specific segmentation data and a user trait set associated with the user profile.

5. The system of claim 4 , wherein the at least one cookie set associated with each user profile comprises a plurality of cookie sets associated with each user profile.

6. The system of claim 1 , wherein allowing the publisher server to access the cookie set comprises allowing the publisher server to update the cookie set.

7. The system of claim 1 , wherein the user profile database stores, for each of the plurality of user profiles, a user trait set, and wherein the at least one processor is further configured to, prior to storing the cookie set of the selected one of the user profiles in the cookie storage of the web crawler, select the one of the user profiles using the user trait set.

8. The system of claim 7 , wherein the at least one processor is further configured to obtain site-specific segmentation data for the publisher web page.

9. The system of claim 8 wherein the at least one processor is further configured to select the one of the user profiles using the user trait set and the obtained site-specific segmentation data.

10. A method, comprising: storing, in a cookie storage of a web crawler, a cookie set associated with a user profile that is stored in a user profile database; with the web crawler, loading a publisher website while allowing the publisher website to access the cookie set in the cookie storage; with advertisement discovery equipment, identifying user-specific advertisements that have been placed on the publisher website, based on the publisher website access to the stored cookie set, by an advertisement channel that is separate from the advertisement discovery equipment, wherein the user-specific advertisements that have been placed on the publisher website are associated with the user profile; and with the advertisement discovery equipment, identifying placement pathways associated with the identified user-specific advertisements that have been placed on the publisher website by the advertisement channel.

11. The method of claim 10 , further comprising selecting the cookie set from a plurality of cookie sets stored with the user profile in the user profile database.

12. The method defined in claim 10 wherein the user profile includes a user trait set and wherein the method further comprises selecting the user profile, from among a plurality of user profiles stored in the user profile database, using the user trait set.

13. The method defined in claim 12 , further comprising obtaining site-specific segmentation data for the publisher website.

14. The method defined in claim 13 wherein selecting the user profile further comprises selecting the user profile using the user trait set and the obtained site-specific segmentation data.

15. The method defined in claim 14 wherein selecting the user profile using the user trait set and the obtained site-specific segmentation data comprises: extracting user population percentages from the obtained site-specific segmentation data; and comparing the extracted user population percentages to the user trait set and additional user trait sets associated with additional ones of the plurality of user profiles in the user profile database.

16. The method defined in claim 10 , further comprising: with data analysis and storage equipment, aggregating advertisement information that includes the identified placement pathways to generate aggregated advertisement information, wherein the aggregated advertisement information comprises at least one of per-service data, per-ad-network data, per-advertiser data, per-creative data, and per-campaign data generated by combining and sorting a plurality of identified advertisements using the data analysis and storage equipment; and with customer interface equipment, providing a notification to a customer that includes the aggregated advertisement information.

17. The method of claim 10 , wherein allowing the publisher website to access the cookie set comprises allowing read and write access to the cookie set in the cookie storage of the web crawler.

18. The method of claim 17 , further comprising updating the cookie set for the user profile in the user profile database after allowing the read and write access to the cookie set in the cookie storage of the web crawler.

19. A method, comprising: loading, with a web crawler, a publisher web page at a publisher server while allowing the publisher server to access a cookie set in a cookie storage of the web crawler, the cookie set associated with a user profile in a user profile database; identifying, using advertisement discovery equipment, a user-specific advertisement that has been placed on the publisher web page, based on the publisher server access to the cookie set, by an advertisement channel that is separate from the advertisement discovery equipment; and with the advertisement discovery equipment, identifying a placement pathway by which the identified user-specific advertisement was placed on the publisher web page by the advertisement channel.

20. The method of claim 19 , further comprising generating the cookie set based on a user trait set associated with the user profile and based on site-specific segmentation data for a plurality of publisher websites, wherein generating the cookie set comprises: selecting the user profile, with cookie harvesting computing equipment, from a plurality of existing user profiles in the user profile database, each user profile corresponding to a simulated interne user and each having a corresponding user trait set; obtaining the site-specific segmentation data for each of the plurality of publisher websites; selecting, from the plurality of publisher websites, a set of publisher websites to browse based on the user trait set for the user profile and the obtained site-specific segmentation data; with the web crawler, loading each of the selected set of publisher websites while allowing each of the selected set of publisher websites to place cookies in the cookie storage of the web crawler; and storing the placed cookies in the user profile database as the cookie set for the user profile.