System and Method for Determining Concepts in a Content Item Using Context

Abstract

The present invention is directed towards systems and methods for indexing one or more items of content. The method of the present invention comprises extracting one or more items of text from a given item of content. The one or more items of extracted text are tokenized into one or more concepts. One or more related concepts associated with the one or more concepts are identified. A support score is generated for the one or more concepts, and the item of content is index with the one or more concepts and the one or more associated support scores.

1 . A method implemented on at least one machine having at least one processor, storage, and a communication platform connected to a network for processing one or more items of content, the method comprising:
 receiving a search query;   identifying, based on an index of items of content, a set of items of content responsive to the search query, wherein the index comprises individual items of content with corresponding one or more concepts and corresponding support scores for each of the one or more concepts, wherein the support score for each concept is determined based on whether that concept appears in the corresponding item of content;   obtaining, for each individual item of content in the set, a sum of the support scores associated with the one or more concepts that are related to the search query; and   providing the set, wherein the items of content in the set are sorted based on the sum of the support scores.   
     
     
         2 . The method of  claim 1 , wherein an item of content comprises a web page, a document, a video file, or an audio file, or a combination thereof. 
     
     
         3 . The method of  claim 1 , determining, for at least one item of content, the corresponding one or more concepts, the determining comprising tokenizing an item of text of the at least one item of content into one or more keywords or phrases. 
     
     
         4 . The method of  claim 1 , further comprising determining the support score for at least one of the one or more concepts for one of the items of content, the determining comprising identifying a frequency with which the at least one concept appears in that one of the items of content. 
     
     
         5 . The method of  claim 3 , wherein tokenizing the item of text into one or more keywords or phrases comprises:
 identifying a frequency with which the one or more keywords or phrases appear in a corpus of content items; and   tokenizing the item of text into the one or more keywords or phrases that appear with the greatest frequency in the corpus of content items.   
     
     
         6 . The method of  claim 1 , further comprising:
 identifying, for at least one item of content, one or more dominant concepts from the one or more concepts, wherein the one or more dominant concepts have corresponding support scores exceeding a support score threshold,   wherein the index comprises the at least one item of content associated with the one or more dominant concepts and the corresponding support scores.   
     
     
         7 . The method of  claim 1 , further comprising identifying, for at least one item of content, one or more related concepts associated with the one or more concepts, the identifying comprising identifying one or more keywords or phrases associated with the one or more concepts,
 wherein the support score for each concept is determined further based on whether the one or more related concepts appear in the corresponding item of content.   
     
     
         8 . A system to process one or more items of content, the system comprising:
 one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the one or more physical processors to implement:
 a concept dictionary operative to maintain concepts related to items of content; and 
 a search provider module configured to:
 receive a search query; 
 identify, based on an index of items of content, a set of items of content responsive to the search query, wherein the index comprises individual items of content with corresponding one or more concepts and corresponding support scores for each of the one or more concepts, wherein the support score for each concept is determined based on whether that concept appears in the corresponding item of content; 
 obtain, for each individual item of content in the set, a sum of the support scores associated with the one or more concepts that are related to the search query; and 
 provide the set, wherein the items of content in the set are sorted based on the sum of the support scores. 
 
   
     
     
         9 . The system of  claim 8 , wherein an item of content comprises a web page, a document, a video file, or an audio file, or a combination thereof. 
     
     
         10 . The system of  claim 8 , wherein the one or more physical processors are further caused to implement: an aboutness extractor configured to determine, for at least one item of content, the corresponding one or more concepts, at least by tokenizing an item of text of the at least one item of content into one or more keywords or phrases. 
     
     
         11 . The system of  claim 8 , wherein the aboutness extractor is further configured to: determine the support score for at least one of the one or more concepts for one of the items of content, at least by identifying a frequency with which the at least one concept appears in that one of the items of content. 
     
     
         12 . The system of  claim 10 , wherein, to tokenize the item of text into one or more keywords or phrases, the aboutness extractor is further configured to:
 identify a frequency with which the one or more keywords or phrases appear in a corpus of content items; and   tokenize the item of text into the one or more keywords or phrases that appear with the greatest frequency in the corpus of content items.   
     
     
         13 . The system of  claim 8 , wherein the search provider module is further configured to:
 identify, for at least one item of content, one or more dominant concepts from the one or more concepts, wherein the one or more dominant concepts have corresponding support scores exceeding a support score threshold,   wherein the index comprises the at least one item of content associated with the one or more dominant concepts and the corresponding support scores.   
     
     
         14 . The system of  claim 10 , wherein the one or more physical processors are further caused to implement: a context dictionary operative to maintain related concepts associated with the concepts maintained in the concept dictionary,
 wherein the aboutness extractor is further configured to:   identify, for at least one item of content, one or more related concepts associated with the one or more concepts, the identifying comprising identifying one or more keywords or phrases associated with the one or more concepts,   wherein the support score for each concept is determined further based on whether the one or more related concepts appear in the corresponding item of content.   
     
     
         15 . A non-transitory computer readable medium storing computer program instructions to process one or more items of content, the computer program instructions that, when executed by one or more physical processors, cause the one or more physical processors to:
 receive a search query;   identify, based on an index of items of content, a set of items of content responsive to the search query, wherein the index comprises individual items of content with corresponding one or more concepts and corresponding support scores for each of the one or more concepts, wherein the support score for each concept is determined based on whether that concept appears in the corresponding item of content;   obtain, for each individual item of content in the set, a sum of the support scores associated with the one or more concepts that are related to the search query; and   provide the set, wherein the items of content in the set are sorted based on the sum of the support scores.   
     
     
         16 . The computer readable medium of  claim 15 , wherein an item of content comprises a web page, a document, a video file, or an audio file, or a combination thereof. 
     
     
         17 . The computer readable medium of  claim 15 , wherein the one or more physical processors are further caused to: determine, for at least one item of content, the corresponding one or more concepts, at least by tokenizing an item of text of the at least one item of content into one or more keywords or phrases. 
     
     
         18 . The computer readable medium of  claim 15 , wherein the one or more physical processors are further caused to: determine the support score for at least one of the one or more concepts for one of the items of content, at least by identifying a frequency with which the at least one concept appears in that one of the items of content. 
     
     
         19 . The computer readable medium of  claim 17 , wherein, to tokenize the item of text into one or more keywords or phrases, the one or more physical processors are further caused to:
 identify a frequency with which the one or more keywords or phrases appear in a corpus of content items; and   tokenize the item of text into the one or more keywords or phrases that appear with the greatest frequency in the corpus of content items.   
     
     
         20 . The computer readable medium of  claim 15 , wherein the one or more physical processors are further caused to: identify, for at least one item of content, one or more related concepts associated with the one or more concepts, the identifying comprising identifying one or more keywords or phrases associated with the one or more concepts,
 wherein the support score for each concept is determined further based on whether the one or more related concepts appear in the corresponding item of content.