Keyword and business tag extraction

Abstract

A system to extract relevant keywords or business tags that describe a company's business is provided. The keyword extraction system utilizes a smart crawler to identify and crawl product pages from a company's website. These pages serve to provide textual descriptions of product offerings, solutions, or services that make up the company's business. The keyword extraction system combines these web documents with other textual descriptions of companies, e.g. from third party data vendors or other public data sources and company databases, to form a corpus of documents that describe companies. The corpus of documents and keywords are processed to segment the plurality of companies into subsets by applying a clustering technique and to provide visualization of the clusters with business tags.

What is claimed is: 
     
         1 . A computer-implemented method, wherein one or more computing devices comprising storage and a processor are programmed to perform steps comprising:
 generating a count for keyword phrases and topics extracted from a corpus of documents, the topics being associated with the extracted keyword phrases or a portion of the extracted keyword phrases;   determining document frequencies (DF) for each extracted keyword phrase across the corpus of documents;
 applying a term-frequency (TF)-inverse-document-frequency (IDF) (TF-IDF) transformation to each of the extracted keyword phrases to generate a respective plurality of TF-IDF vectors; 
   determining a strength of each topic based on a number of extracted keyword phrases associated with that respective topic;
 determining an edge weight based on a linkage of the topic with an associated extracted keyword phrase; 
   generating relevance scores relating each extracted keyword phrase to the respective company based on a strength of each of the extracted keyword phrases, the strength of each topic, and the edge weight for each topic, the strength of each of the extracted keyword phrases being equal to a TF-IDF vector associated with the extracted keyword phrase;   applying a representation learning technique to the plurality of TF-IDF vectors and the relevance scores to generalize each respective company into at least one of a plurality of topic spaces;   segmenting the plurality of companies into clusters by applying a clustering technique to the extracted keyword phrases for each respective company or to the plurality of topic spaces; and   outputting the clusters of companies with respective business tags.   
     
     
         2 . The method of  claim 1 , further comprising generating similarity between two of the extracted keyword phrases based on a distance metric between the two extracted keyword phrases and determining the strength of each topic based on the number of extracted keyword phrases and the similarity between the two extracted keyword phrases. 
     
     
         3 . The method of  claim 2 , wherein the distance metric includes a distance between the two extracted keyword phrases as either cosine distance or Euclidian distance. 
     
     
         4 . The method of  claim 1 , further comprising generating similarity between two of the extracted keyword phrases based on a positive point-wise mutual information (PPMI) matrix of the two extracted keyword phrases to context words and determining the strength of each topic based on the number of extracted keyword phrases and the similarity between the two extracted keyword phrases. 
     
     
         5 . The method of  claim 4 , further comprising segregating the context words by regions of distances away from a central keyword phrase. 
     
     
         6 . The method of  claim 4 , further comprising generating a co-occurrence matrix of the two extracted keyword phrases to context words by counting the occurrences of each pair of (w, c), wherein w is the extracted keyword phrase and c is a context word within a specific zone. 
     
     
         7 . The method of  claim 1 , further comprising segmenting the plurality of companies into a first cluster and a second, overlapping cluster. 
     
     
         8 . The method of  claim 1 , further comprising segmenting the plurality of companies into a first cluster and a second, non-overlapping cluster. 
     
     
         9 . The method of  claim 1 , further comprising segmenting the plurality of companies into a first cluster and a second cluster that is larger than the first cluster. 
     
     
         10 . The method of  claim 1 , further comprising segmenting the plurality of companies into a first cluster and a second cluster that is approximately the same size as the first cluster. 
     
     
         11 . The method of  claim 1 , further comprising segmenting the plurality of companies into a first cluster and a second cluster, and extracting keywords for each of the first cluster and the second cluster. 
     
     
         12 . The method of  claim 11 , further comprising generating the relevance scores relating to the first cluster and the second cluster based on a strength of each of the extracted keywords for the first cluster and the second cluster, respectively, and outputting the relevance scores relating to the first cluster and the second cluster. 
     
     
         13 . A system, comprising:
 a processor configured to:   generate a count for keyword phrases and topics extracted from a corpus of documents, the topics being associated with the extracted keyword phrases or a portion of the extracted keyword phrases;   determine document frequencies (DF) for each extracted keyword phrase across the corpus of documents;   apply a term-frequency (TF)-inverse-document-frequency (IDF) (TF-IDF) transformation to each of the extracted keyword phrases to generate a respective plurality of TF-IDF vectors;   determine a strength of each topic based on a number of extracted keyword phrases associated with that respective topic;   determine an edge weight based on a linkage of the topic with an associated extracted keyword phrase;   generate relevance scores relating each extracted keyword phrase to the respective company based on a strength of each of the extracted keyword phrases, the strength of each topic, and the edge weight for each topic, the strength of each of the extracted keyword phrases being equal to a TF-IDF vector associated with the extracted keyword phrase;   apply a representation learning technique to the plurality of TF-IDF vectors and the relevance scores to generalize each respective company into at least one of a plurality of topic spaces;   create segments of the plurality of companies into clusters by applying a clustering technique to the extracted keyword phrases for each respective company or to the plurality of topic spaces; and   an output configured to transmit the clusters of companies with respective business tags to another computing device, network, or system.   
     
     
         14 . The system of  claim 13 , wherein the processor is further comprised to generate similarity between two of the extracted keyword phrases based on a distance metric between the two extracted keyword phrases and determine the strength of each topic based on the number of extracted keyword phrases and the similarity between the two extracted keyword phrases. 
     
     
         15 . The system of  claim 13 , wherein the processor is further configured to generate similarity between two of the extracted keyword phrases based on a positive point-wise mutual information (PPMI) matrix of the two extracted keyword phrases to context words and determine the strength of each topic based on the number of extracted keyword phrases and the similarity between the two extracted keyword phrases. 
     
     
         16 . The system of  claim 15 , wherein the processor is further configured to segregate the context words by regions of distances away from a central keyword phrase. 
     
     
         17 . The system of  claim 15 , wherein the processor is further configured to generate a co-occurrence matrix of the two extracted keyword phrases to context words by counting the occurrences of each pair of (w, c), wherein w is the extracted keyword phrase and c is a context word within a specific zone. 
     
     
         18 . The system of  claim 13 , wherein the processor is further configured to create segments of the plurality of companies into a first cluster and a second, overlapping cluster or a first cluster and a second, non-overlapping cluster. 
     
     
         19 . The system of  claim 13 , wherein the processor is further configured to create segments of the plurality of companies into a first cluster and a second cluster that is larger than the first cluster or approximately the same size as the first cluster. 
     
     
         20 . The system of  claim 13 , wherein:
 the processor is further configured to:   create segments of the plurality of companies into a first cluster and a second cluster,   extract keywords for each of the first cluster and the second cluster, and   generate t h e relevance scores relating to the first cluster and the second cluster based on a strength of each of the extracted keywords for the first cluster and the second cluster, respectively, and   the output is further configured to output the relevance scores relating to the first cluster and the second cluster.