Search Engine Data Structure

Abstract

A search engine database may a segmented structure that preserves individual document references and allows updating as well as scalability. A set of segment managers may receive new, updated, or deleted documents and update a set of term matrices from which a published search matrix may be generated. The database may have a very large term dictionary and may use a hash function to create term identifiers without having to look up terms in the dictionary. The database may be maintained by many systems operating in parallel for high scalability.

What is claimed is: 
     
         1 . A method performed on at least one computer processor, the method comprising:
 receiving a first document comprising tags and words;   creating a storage document comprising term identifiers by a first method comprising:
 for each of the tags, determining a term for a tag, and determining a term identification by performing a hash on the term, replacing the tag with the term identifier in the storage document; 
 for each of the words, determining a term identifier for the word and replacing the word with the term identifier in the storage document; and 
 storing the storage document. 
   
     
     
         2 . The method of  claim 1  further comprising:
 identifying a plurality of ngrams in the first document; and 
 storing the ngrams in the storage document. 
 
     
     
         3 . The method of  claim 2  wherein the ngrams are a three letter trigrams. 
     
     
         4 . The method of  claim 1  wherein the words comprise words and phrases. 
     
     
         5 . The method of  claim 1  wherein the term identifier for the words is determined by looking up the word in a word dictionary. 
     
     
         6 . The method of  claim 1  wherein the hash is an MD5 hash. 
     
     
         7 . The method of  claim 1  further comprising:
 identifying as at least a portion of the tags items in the first document that correspond to items that can be used to search the first document 
 
     
     
         8 . The method of  claim 1  further comprising identifying words in the first document. 
     
     
         9 . The method of  claim 8  wherein determining a term identifier for the words is performed on a subset of the words identified in the first document. 
     
     
         10 . A system comprising:
 a collector connected to at least one data source configured to identify documents in the at least one data source, the documents including words and terms;   a receiver configured to receive the identified documents from the collector and assign each received document a document identifier;   a preprocessor configured to prepare the identified documents for indexing and searching;   an indexer that identifies words within at least one of the identified documents and determines a term identifier for each identified word, and replaces the words with the determined term identifier for each word;   a term processor that identifies within the at least one of the identified documents and determines a term identifier for the identified terms, and replaces the terms with the determined term identifier for the identified term; and   a distributor that identifies a segment manager from a plurality of segment managers based on the document identifier for the received document and transmits at least the document identifier to the identified segment manager.   
     
     
         11 . The system of  claim 10  wherein each of the plurality of segment managers is configured to create a plurality of individual matrices for the received document. 
     
     
         12 . The system of  claim 10  further comprising:
 a plurality of buffers, the plurality of buffers disposed between at least two of the collector, the receive, the preprocessor, the indexer, the term processor and the distributor. 
 
     
     
         13 . The system of  claim 10  wherein the collector is a plurality of collectors. 
     
     
         14 . The system of  claim 10  wherein the indexer determines a term identifier for a subset of the identified words. 
     
     
         15 . The system of  claim 10  further comprising:
 a collection manager configured to monitor the distributor for documents and to store information regarding the documents in a modification database; and 
 the modification database including at least the document identifier for the document and a timestamp indicating when the document was provided to the distributor. 
 
     
     
         16 . A method performed on at least one computer processor, the method comprising:
 creating a plurality of search matrices from a storage document, at least one of the plurality of search matrices being an individual term search matrix;   gathering a plurality of the individual term search matrices together into a consolidated term search matrix, each of the plurality of the individual term search matrices having been created from a different document;   receiving a search request comprising a first tag;   creating a first tag identifier from the first tag;   searching for the first tag identifier as a term in the consolidated term search matrix to identify the first document; and   presenting the first document as a search result.   
     
     
         17 . The method of  claim 16  wherein creating a first tag identifier further comprises:
 performing a hash on the first tag; and replacing the first tag with the hashed version of the first tag. 
 
     
     
         18 . The method of  claim 17  wherein the hash is an MD5 hash. 
     
     
         19 . The method of  claim 16  wherein creating a first tag identifier further comprises:
 querying a look-up table to determine the first tag identifier. 
 
     
     
         20 . The method of  claim 16  wherein creating a first tag identifier further comprises:
 querying a dictionary to determine the first tag identifier.