Inferring information about a webpage based upon a uniform resource locator of the webpage

Abstract

Described herein are technologies related to inferring information about a webpage based upon semantics of a uniform resource location (URL) of the webpage. The URL is tokenized to create a sequence of tokens. An embedding for the URL is generated based upon the sequence of tokens, wherein the embedding is representative of semantics of the URL. Based upon the embedding for the URL, information about the webpage pointed to by the URL is inferred, the webpage is retrieved, and information is extracted from the webpage based upon the information inferred about the webpage.

What is claimed is: 
     
         1 . A computing system comprising:
 a processor; and   memory storing instructions that, when executed by the processor, cause the processor to perform acts comprising:
 obtaining a uniform resource locator (URL) of a webpage, where the URL includes a sequence of alphanumerical characters; 
 creating an embedding for the URL of the webpage, where the embedding for the URL comprises a numeric semantic representation of the sequence of alphanumerical characters in the URL; 
 providing the embedding for the URL to a computer-implemented classifier, where the computer-implemented classifier identifies that the webpage includes content pertaining to a topic based upon the embedding for the URL, where the computer-implemented classifier identifies that the webpage includes the content pertaining to the topic without retrieving the content from the webpage; and 
 based upon the computer-implemented classifier identifying that the webpage includes the content pertaining to the topic, assigning metadata to the URL in computer-readable storage, where the metadata indicates that the webpage includes the content pertaining to the topic. 
   
     
     
         2 . The computing system of  claim 1 , where the webpage is not represented in an index of a search engine. 
     
     
         3 . The computing system of  claim 1 , where creating the embedding for the URL of the webpage comprises:
 tokenizing the sequence of alphanumerical characters to create a sequence of tokens; and   identifying a sequence of numerical identifiers that corresponds to the sequence of tokens, where the embedding for the URL is created based upon the sequence of numerical identifiers.   
     
     
         4 . The computing system of  claim 3 , where creating the embedding for the URL further comprises:
 performing word embedding on the sequence of numerical identifiers to form a sequence of s-dimensional vectors, where the embedding for the URL is created based upon the sequence of s-dimensional vectors.   
     
     
         5 . The computing system of  claim 1 , the acts further comprising:
 obtaining a second URL of a second webpage, where the second URL includes a second sequence of alphanumerical characters;   creating a second embedding for the second URL of the second webpage, where the second embedding for the second URL comprises a second numeric semantic representation of the second sequence of alphanumerical characters in the second URL;   providing the second embedding for the second URL to a second computer-implemented classifier, where the second computer-implemented classifier identifies that the second webpage is likely to cause a client computing device to install malware on the client computing device upon the client computing device retrieving the second webpage; and   based upon the second computer-implemented classifier identifying that the second webpage is likely to cause the client computing device to install malware on the client computing device upon the client computing device retrieving the second webpage, assigning second metadata to the second URL in the computer-readable storage, where the second metadata indicates that the second webpage is associated with malware.   
     
     
         6 . The computing system of  claim 1 , the acts further comprising:
 obtaining a second URL of a second webpage, where the second URL includes a second sequence of alphanumerical characters;   creating a second embedding for the second URL of the second webpage, where the second embedding for the second URL comprises a second numeric semantic representation of the second sequence of alphanumerical characters in the second URL;   providing the second embedding for the second URL to a second computer-implemented classifier, where the second computer-implemented classifier identifies that the second webpage likely includes content written in a specific language; and   based upon the second computer-implemented classifier identifying that the second webpage is likely to include the content written in the specification language, assigning second metadata to the second URL in the computer-readable storage, where the second metadata indicates that the second webpage includes the content written in the second language.   
     
     
         7 . The computing system of  claim 1 , the acts further comprising:
 obtaining a second URL of a second webpage, where the second URL includes a second sequence of alphanumerical characters;   creating a second embedding for the second URL of the second webpage, where the second embedding for the second URL comprises a second numeric semantic representation of the second sequence of alphanumerical characters in the second URL;   providing the second embedding for the second URL to a second computer-implemented classifier, where the second computer-implemented classifier identifies that the second webpage likely includes at least a threshold number of outbound links; and   based upon the second computer-implemented classifier identifying that the second webpage is likely to include at least the threshold number of output links, assigning second metadata to the second URL in the computer-readable storage, where the second metadata indicates that the second webpage includes at least the threshold number of outbound links.   
     
     
         8 . The computing system of  claim 1 , wherein the computer-implemented classifier is trained based upon content of a search log of a search engine, wherein training data for training the classifier includes URLs in the search log of the search engine and indications as to topics that correspond to webpages pointed to by the URLs. 
     
     
         9 . The computing system of  claim 1 , the acts further comprising:
 obtaining a second URL of a second webpage, where the second URL includes a second sequence of alphanumerical characters;   creating a second embedding for the second URL of the second webpage, where the second embedding for the second URL comprises a second numeric semantic representation of the second sequence of alphanumerical characters in the second URL;   providing the second embedding for the second URL to a second computer-implemented classifier, where the second computer-implemented classifier generates an output that indicates that the second webpage will likely be selected by an arbitrary user of a search engine if the second webpage were represented on a search engine results page (SERP) returned to the arbitrary user; and   based upon the second computer-implemented classifier generating the output, updating a search engine index to include a reference to the second webpage.   
     
     
         10 . The computing system of  claim 1 , the acts further comprising:
 obtaining a second URL of a second webpage, where the second URL includes a second sequence of alphanumerical characters;   creating a second embedding for the second URL of the second webpage, where the second embedding for the second URL comprises a second numeric semantic representation of the second sequence of alphanumerical characters in the second URL;   providing the second embedding for the second URL to a second computer-implemented classifier, where the second computer-implemented classifier generates an output that indicates that the second webpage has likely been updated within a threshold amount of time; and   based upon the second computer-implemented classifier generating the output, retrieving the second webpage and extracting content therefrom in connection with updating a search engine index.   
     
     
         11 . A method executed by at least one processor of a computing system, the method comprising:
 retrieving a uniform resource locator (URL) for a webpage from a list of URLs for webpages, wherein the webpage is included in the World Wide Web;   creating, based upon the URL, a vector of values that represents semantics existent in alphanumerical characters of the URL;   determining that the webpage likely pertains to a topic based upon the vector of values that represents the semantics existent in the alphanumerical characters of the URL; and   based upon the determining that the webpage likely pertains to the topic, assigning metadata to the URL in computer-readable storage, where the metadata indicates that the webpage pertains to the topic.   
     
     
         12 . The method of  claim 11 , wherein determining that the webpage likely pertains to the topic comprises providing the vector of values to a computer-implemented classifier, where the computer-implemented classifier outputs an indication that the webpage likely pertains to the topic. 
     
     
         13 . The method of  claim 11 , wherein creating the vector of values that represents semantics existent in the alphanumerical characters of the URL comprises:
 tokenizing the URL to extract tokens from the URL;   mapping the extracted tokens to respective identifiers;   generating n-grams from the extracted tokens, wherein each n-gram includes several tokens; and   using word embedding, and based upon the n-grams, generating s-dimensional vectors for the n-grams, wherein the s-dimensional vectors represent semantics of the n-grams.   
     
     
         14 . The method of  claim 13 , wherein the s-dimensional vectors are  2 -dimensional vectors. 
     
     
         15 . The method of  claim 11 , further comprising:
 retrieving a second URL for a second webpage from the list of URLs for webpages, wherein the second webpage is included in the World Wide Web;   creating, based upon the second URL, a second vector of values that represents second semantics existent in second alphanumerical characters of the second URL;   determining that the second webpage likely includes malware based upon the second vector of values that represents the second semantics existent in the second alphanumerical characters of the second URL; and   based upon the determining that the second webpage likely includes malware, assigning second metadata to the second URL in the computer-readable storage, where the second metadata indicates that the second webpage includes malware.   
     
     
         16 . The method of  claim 11 , further comprising:
 retrieving a second URL for a second webpage from the list of URLs for webpages, wherein the second webpage is included in the World Wide Web;   creating, based upon the second URL, a second vector of values that represents second semantics existent in second alphanumerical characters of the second URL;   determining that the second webpage likely includes at least a threshold number of outbound links based upon the second vector of values that represents the second semantics existent in the second alphanumerical characters of the second URL; and   based upon the determining that the second webpage likely includes at least the threshold number of outbound links, assigning second metadata to the second URL in the computer-readable storage, where the second metadata indicates that the second webpage includes at least the threshold number of outbound links.   
     
     
         17 . The method of  claim 11 , further comprising:
 retrieving a second URL for a second webpage from the list of URLs for webpages, wherein the second webpage is included in the World Wide Web;   creating, based upon the second URL, a second vector of values that represents second semantics existent in second alphanumerical characters of the second URL;   determining that the second webpage likely includes content written in a specific language based upon the second vector of values that represents the second semantics existent in the second alphanumerical characters of the second URL; and   based upon the determining that the second webpage likely includes the content in the specific language, assigning second metadata to the second URL in the computer-readable storage, where the second metadata indicates that the second webpage includes content in the specific language.   
     
     
         18 . The method of  claim 11 , further comprising:
 retrieving a second URL for a second webpage from the list of URLs for webpages, wherein the second webpage is included in the World Wide Web;   creating, based upon the second URL, a second vector of values that represents second semantics existent in second alphanumerical characters of the second URL;   determining that the second webpage likely is a permission-based webpage based upon the second vector of values that represents the second semantics existent in the second alphanumerical characters of the second URL; and   based upon the determining that the second webpage likely includes the content in the specific language, assigning second metadata to the second URL in the computer-readable storage, where the second metadata indicates that the second webpage is a permission-based webpage.   
     
     
         19 . The method of  claim 11 , further comprising:
 retrieving a second URL for a second webpage from the list of URLs for webpages, wherein the second webpage is included in the World Wide Web;   creating, based upon the second URL, a second vector of values that represents second semantics existent in second alphanumerical characters of the second URL;   determining that the second webpage likely is unable to be successfully crawled by a web crawler based upon the second vector of values that represents the second semantics existent in the second alphanumerical characters of the second URL; and   based upon the determining that the second webpage likely includes the content in the specific language, assigning second metadata to the second URL in the computer-readable storage, where the second metadata indicates that the second webpage is unable to be crawled by the web crawler.   
     
     
         20 . A computer-readable storage medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform acts comprising:
 retrieving a uniform resource locator (URL) for a webpage from a list of URLs for webpages, wherein the webpage is included in the World Wide Web;   creating, based upon the URL, a vector of values that represents semantics existent in alphanumerical characters of the URL;   determining that the webpage likely pertains to a topic based upon the vector of values that represents the semantics existent in the alphanumerical characters of the URL; and   based upon the determining that the webpage likely pertains to the topic, assigning metadata to the URL in computer-readable storage, where the metadata indicates that the webpage pertains to the topic.