Identifying and evaluating online references

Abstract

One example embodiment includes a method for indexing online references of an entity. The method includes identifying one or more channels of the Internet to be searched for references to an entity and identifying one or more signals to be evaluated within each of the one or more channels. The method also includes crawling the Internet for online references to the entity, wherein crawling the Internet comprises searching the one or more channels of the Internet for references to the entity and evaluating the one or more signals. The method further includes constructing a reverse index of the references, wherein the reverse index is based on each channel in which a reference is found and the one or more signals evaluated for the reference.

What is claimed is: 
     
       1. A method of indexing online references of an entity, the method comprising:
 identifying an entity, wherein the entity is an individual, corporation, brand, or product;   crawling the Internet for online references to the entity after identifying the entity, wherein crawling the Internet comprises:
 querying a first search engine for a first search results page using a keyword; 
 parsing the first search result pages into organic search results; 
 identifying an organic online reference to the entity based on the organic search results; 
 parsing the organic search results to identify one or more organic signals that include information about the organic online reference to the entity; 
 searching social networks for social network search results that refer to the entity; 
 identifying a social online reference to the entity based on the social network search results; 
 parsing the social network search results to identify one or more social signals that include information about the social online reference to the entity; 
 evaluating the social signals and organic signals; and 
   constructing, using a processor, a reverse index of the online references based on the evaluated social signals and the evaluated organic signals.   
     
     
       2. The method of  claim 1 , further comprising computing a search engine optimization score for the identified organic online reference to the entity based on the reverse index. 
     
     
       3. The method of  claim 1 , wherein the reverse index lists the social online reference and the organic online reference. 
     
     
       4. The method of  claim 1 , wherein the reverse index lists the organic online reference with respect to the keyword. 
     
     
       5. The method of  claim 1 , wherein the social signals comprise a rank, a uniform resource locator (URL), a title, or a description of the social online reference. 
     
     
       6. The method of  claim 1 , wherein the social signals are evaluated to identify the relevance of the social online reference to the entity. 
     
     
       7. The method of  claim 1 , wherein the organic signals comprise a rank, a uniform resource locator (URL), a title, or a description of the organic online reference. 
     
     
       8. The method of  claim 1 , wherein the organics signals are evaluated to identify the relevance of the organics online reference to the entity. 
     
     
       9. The method of  claim 1 , wherein the social networks comprise a social structure of nodes that are tied by one or more specific types of interdependency. 
     
     
       10. The method of  claim 1 , wherein the organic search results are search results in a search engine results page that appear based on their relevance to a search term used to generate the search results. 
     
     
       11. A method of indexing online references of an entity, the method comprising, by a system comprising a plurality of worker nodes:
 crawling the Internet for the online references to the entity by querying, using a keyword, one or more search engines to retrieve search results pages,
 wherein the entity is an individual, corporation, brand, or product, 
 wherein the search results pages comprise one or more search results identified by the respective search engine as responsive to the keyword, and 
 wherein the worker nodes are configured to simulate activities of a human user of the Internet by providing the one or more search engines with information that affects how the search results would be presented to the human user; 
   parsing the search results pages to identify organic search results, wherein the organic search results appear in the search results pages based on relevance of webpages corresponding to the organic search results to the keyword;   parsing the organic search results to identify organic online references to entities in the organic search results and one or more signals to be evaluated that include information about the organic online references;   evaluating the signals to identify relevance of the organic online references to the entities in the organic search results;   constructing a reverse index of the online references based on the evaluated signals;   in response to receiving a query related to the entity, computing, using the reverse index, statistics regarding the organic online references to the identified entity, wherein computing the statistics comprises attributing an organic online reference to the identified entity identified via a first search engine of the one or more search engines with a weight greater than an organic online reference to the entity identified via a second search engine; and   presenting the computed statistics regarding the organic online references to the identified entity.   
     
     
       12. The method of claim 11, wherein the information that affects how the search results would be presented to the human user comprises:
 the keyword;   attributes of the human user;   attributes of a computing device used by the human user to perform the activities; or   contextual information related to the crawling.   
     
     
       13. The method of claim 12, wherein the information that affects how the search results would be presented to the human user comprises a geographic location, demographic information of the human user, group characteristics of the human user, software installed on the computing device, or technical features of the computing device. 
     
     
       14. The method of claim 11, wherein the evaluated signals comprise placement of the organic online references in the search results pages, contextual information related to the crawl, time to load a web page, contextual information related to the organic search results, contextual information related to the organic online references, technical information related to the organic online references, technical information related to the organic search results, or the information provided during the crawl that affects how the search results would be presented to the human user. 
     
     
       15. The method of claim 11, wherein organic search results are associated with different categories, and wherein each of the categories represents a type of online reference. 
     
     
       16. The method of claim 15, wherein the type of online reference comprises one of web page content, advertisements, content referenced in social networks, social news, content referenced in discussion forums or blogs, content referenced in news sites, rich media, images, video, social bookmarks, game content, and subtypes thereof. 
     
     
       17. The method of claim 11, wherein the computing the statistics regarding the organic online references to the identified entity further comprises computing a number of hits for a web page containing the organic online reference to the identified entity, statistics related to mentions of the identified entity within a particular type of the organic online references, a click analysis based on a number of users that follow a link to a uniform resource locator (URL), or a characteristic of the mentions of the identified entity. 
     
     
       18. The method of claim 11, wherein the computing the statistics regarding the organic online references to the identified entity further comprises computing a search engine optimization score for the identified entity based on the organic online references in the reverse index. 
     
     
       19. The method of claim 11, wherein the computing the statistics regarding the organic online references to the identified entity further comprises identifying backlinks to the organic online references based on the search results. 
     
     
       20. The method of claim 11, further comprising performing a trend analysis to determine how the organic online references have changed over time. 
     
     
       21. A method of indexing online references of an entity, the method comprising, by a system comprising a plurality of worker nodes:
 crawling the Internet for the online references to the entity by querying, using a keyword, one or more search engines to retrieve search results pages,
 wherein the entity is an individual, corporation, brand, or product, 
 wherein the search results pages comprise one or more search results identified by the respective search engine as responsive to the keyword, and 
 wherein the worker nodes are configured to simulate the activities of a human user of the Internet by providing the one or more search engines with information that affects how the search results would be presented to the human user; 
   parsing the search results pages to identify paid content based on placement or inclusion;   parsing the paid content to identify paid online references to entities and one or more signals to be evaluated that include information about the paid online references;   evaluating the signals to identify relevance of the paid online references to the entities;   constructing a reverse index of the online references based on the evaluated signals;   in response to receiving a query related to the entity, computing, using the reverse index, statistics regarding the paid online references to the identified entity, wherein computing the statistics comprises attributing a paid online reference to the identified entity identified via a first search engine of the one or more search engines with a weight greater than a paid online reference to the entity identified via a second search engine; and   presenting the computed statistics regarding the paid online references to the identified entity.   
     
     
       22. The method of claim 21, wherein the information that affects how the search results would be presented to the human user comprises:
 the keyword;   attributes of the human user;   attributes of a computing device used by the human user to perform the activities; or   contextual information related to the crawling.   
     
     
       23. The method of claim 22, wherein the information that affects how the search results would be presented to the human user comprises a geographic location, demographic information of the human user, group characteristics of the human user, software installed on the computing device, or technical features of the computing device. 
     
     
       24. The method of claim 21, wherein parsing the search results pages to identify paid content further comprises parsing the search results pages to identify results associated with different categories, and wherein each of the categories represents a type of online reference. 
     
     
       25. The method of claim 24, wherein the type of online reference comprises one of web page content, advertisements, content referenced in social networks, social news, content referenced in discussion forums or blogs, content referenced in news sites, rich media, images, video, social bookmarks, game content, and subtypes thereof. 
     
     
       26. The method of claim 21, wherein the evaluated signals placement of the paid online references in the search results pages, contextual information related to the crawl, contextual information related to the search results, contextual information related to the paid online references, technical information related to the paid online references, technical information related to the search results, or the information provided during the crawl that affects how the search results would be presented to the human user. 
     
     
       27. The method of claim 21, wherein the computing the statistics regarding the paid online references to the identified entity further comprises computing a paid search optimization score for the identified entity based on the paid online references in the reverse index. 
     
     
       28. The method of claim 21, wherein the computing the statistics regarding the paid online references to the identified entity further comprises computing a number of hits for a web page containing the paid online reference to the identified entity, a number of mentions of the identified entity within a particular type of the paid online references, a click analysis based on a number of users that follow a link to a uniform resource locator (URL), or a characteristic of the mentions of the identified entity. 
     
     
       29. The method of claim 21, wherein the computing the statistics regarding the paid online references to the identified entity further comprises identifying backlinks to the paid online references based on the search results pages. 
     
     
       30. The method of claim 21, further comprising performing a trend analysis to determine how the paid online references have changed over time. 
     
     
       31. A method of indexing online references of an entity, the method comprising, by a system comprising a plurality of worker nodes:
 crawling the Internet for the online references to the entity by querying, using a keyword, one or more search engines to retrieve search results pages,
 wherein the entity is an individual, corporation, brand, or product, 
 wherein the search results pages comprise one or more search results identified by the respective search engine as responsive to the keyword, and 
 wherein the worker nodes are configured to simulate the activities of a human user of the Internet by providing the one or more search engines with information that affects how the search results would be presented to the human user; 
   parsing the search results pages to identify organic search results and paid content, wherein the organic search results appear in the search results pages based on relevance of webpages corresponding to the organic search results to the keyword, and wherein the paid content appears in the search results pages based on placement or inclusion;   parsing the organic search results to identify organic online references to entities in the organic search results and one or more organic signals to be evaluated that include information about the organic online references;   parsing the paid content to identify paid online references to entities and one or more paid signals to be evaluated that include information about the paid online references;   evaluating the organic signals and the paid signals to identify relevance of the organic online references and relevance of the paid online references to the entities in the search results;   constructing a reverse index of the online references based on the evaluated signals;   in response to receiving a query related to the entity, computing, using the reverse index, statistics regarding the organic online references to the identified entity and statistics regarding the paid online references to the identified entity,
 wherein computing the statistics regarding the organic online references comprises attributing an organic online reference to the identified entity identified via a first search engine of the one or more search engines with a first weight greater than an organic online reference to the entity identified via a second search engine, and 
 wherein computing the statistics regarding the paid online references comprises attributing a paid online reference to the identified entity identified via the first search engine of the one or more search engines with a second weight greater than a paid online reference to the entity identified via the second search engine; and 
   presenting the computed statistics regarding the organic online references to the identified entity and the computed statistics regarding the paid online references to the identified entity.   
     
     
       32. The method of claim 31, wherein the information that affects how the search results would be presented to the human user comprises:
 the keyword;   attributes of the human user;   attributes of a computing device used by the human user to perform the activities; or   contextual information related to the crawling.   
     
     
       33. The method of claim 32, wherein the information that affects how the search results would be presented to the human user comprises a geographic location, demographic information of the human user, group characteristics of the human user, software installed on the computing device, or technical features of the computing device. 
     
     
       34. The method of claim 31, wherein the organic search results and the paid content are associated with different categories, and wherein each of the categories represents a type of online reference. 
     
     
       35. The method of claim 34, wherein the type of online reference comprises one of web page content, advertisements, content referenced in social networks, social news, content referenced in discussion forums or blogs, content referenced in news sites, rich media, images, video, social bookmarks, game content, and subtypes thereof. 
     
     
       36. The method of claim 35, wherein the type of the organic online references comprises either content related to social networks or blogs, and wherein the computed statistics comprises a number of mentions of the identified entity, a number of a particular different type of mentions of the identified entity, or a comparative measure of the number of mentions of the identified entity with respect to a second identified entity. 
     
     
       37. The method of claim 31, wherein the computing the statistics regarding the organic online references to the identified entity further comprises computing a number of hits for a web page containing the organic online reference to the identified entity, statistics related to mentions of the identified entity within a particular type of the organic online references, a click analysis based on a number of users that follow a link to a uniform resource locator (URL), or a characteristic of the mentions of the identified entity. 
     
     
       38. The method of claim 31, wherein the computing the statistics regarding the organic online references to the identified entity and the statistics regarding the paid online references to the identified entity further comprises determining a particular weight to be applied to a particular type of the online references or a particular grouping of types of the online references. 
     
     
       39. The method of claim 31, wherein parsing the search results pages to identify organic search results and paid content further comprises parsing the search results pages to identify results associated with different categories, and wherein each of the categories represents a type of online reference. 
     
     
       40. The method of claim 39, wherein the type of online reference comprises one of web page content, advertisements, content referenced in social networks, social news, content referenced in discussion forums or blogs, content referenced in news sites, rich media, images, video, social bookmarks, or game content. 
     
     
       41. The method of claim 31, wherein the evaluated signals comprise contextual information related to the crawl, time to load a web page, placement of the organic online references in the search results pages, contextual information related to the organic search results, contextual information related to the organic online references, technical information related to the organic online references, technical information related to the organic search results, placement of the paid online references in the search results pages, contextual information related to the paid content, contextual information related to the paid online references, technical information related to the paid content, technical information related to the paid online references, or the information provided during the crawl that affects how the search results would be presented to the human user. 
     
     
       42. The method of claim 31, wherein the computing the statistics regarding the organic online references to the identified entity further comprises computing a search engine optimization score for the identified entity based on the organic online references in the reverse index. 
     
     
       43. The method of claim 31, wherein the computing the statistics regarding the paid online references to the identified entity further comprises computing a search engine optimization score for the identified entity based on the paid online references in the reverse index. 
     
     
       44. The method of claim 31, wherein the computing the statistics regarding the organic online references to the identified entity further comprises computing a number of hits for a web page containing the organic online reference to the identified entity, a number of mentions of the identified entity in a channel-related grouping, a click analysis based on a number of users that follow a link to a uniform resource locator (URL), or a characteristic of the mentions of the identified entity. 
     
     
       45. The method of claim 31, wherein the computing the statistics regarding the paid online references to the identified entity further comprises computing a number of hits for a web page containing the paid online reference to the identified entity, a number of mentions of the identified entity within a particular type of the paid online references, a click analysis based on a number of users that follow a link to a uniform resource locator (URL), or a characteristic of the mentions of the identified entity. 
     
     
       46. The method of claim 31, wherein the computing the statistics regarding the organic online references to the identified entity further comprises identifying backlinks to the organic online references based on the search results. 
     
     
       47. The method of claim 31, wherein the computing the statistics regarding the paid online references to the identified entity further comprises identifying backlinks to the paid online references based on the search results. 
     
     
       48. The method of claim 31, further comprising performing a trend analysis to determine how the organic online references and the paid online references have changed over time.