Methods and apparatus for generating a data dictionary

Abstract

Systems and methods for generating data dictionaries are disclosed. In one example, a system comprises a candidate list generator module to generate a list of keywords from search query information, a validation module to validate candidate token pairs based on a divergence value wherein candidate token pairs having divergence values exceeding a divergence threshold are valid token pairs, and a data dictionary module to receive the validated token pairs as entries in a vocabulary, and provide the validated token pairs in response to a search query containing a keyword corresponding to at least one entry in the vocabulary.

1 . An ecommerce system having a database comprising multiple data items, wherein the data items represent products sold in the ecommerce system, the system comprising:
 a processor-implemented candidate list generator module to:
 generate a list of keywords from search query information; 
 generate a set of token pairs including a keyword from the list of keywords and a token, the token being a synonym of the keyword; 
 receive demand information and supply information, wherein the demand information is information retrieved from query logs maintained for user-provided query entries stored in the database and the supply information is seller-provided descriptor information for the data items stored in the database; and 
 apply candidate selection rules to token pairs across multiple categories using the demand information; 
   a processor-implemented validation module to:
 calculate a divergence value for each token pair across multiple categories using the demand information; and 
 validate token pairs based on the divergence value; and 
   a processor-implemented data dictionary module to:
 receive the validated token pairs as entries in a vocabulary, and 
 provide the token pairs in response to a search query containing a keyword corresponding to at least one entry in the vocabulary. 
   
     
     
         2 . A system comprising:
 a processor;   a processor-implemented candidate list generator module to:
 generate a list of keywords from search query information; 
 generate a set of token pairs including a keyword from the list of keywords and a token, the token being a synonym of the keyword; 
 receive demand information and supply information, wherein the demand information is information retrieved from query logs maintained for user-provided query entries stored in the database and the supply information is seller-provided descriptor information for the data items stored in the database; and 
 apply candidate selection rules to token pairs across a plurality of categories of data items in the database using the demand information and the supply information, generating a set of candidate token pairs; 
   a processor-implemented validation module to:
 validate the candidate token pairs based on a divergence value, wherein the candidate token pairs having divergence values exceeding a divergence threshold are valid token pairs; and 
   a processor-implemented data dictionary module to:
 receive the validated token pairs as entries in a vocabulary, and provide the validated token pairs in response to a search query containing a keyword corresponding to at least one entry in the vocabulary. 
   
     
     
         3 . The system of  claim 2 , wherein the candidate list generator module comprises candidate selection rules, including vocabulary generation rules for processing search query terms to identify keywords, and token generation rules for generating synonyms of the keywords. 
     
     
         4 . The system of  claim 2 , wherein the candidate list generator module is further to generate a set of token pairs resulting in a plurality of token pairs having a same keyword. 
     
     
         5 . The system of  claim 2 , wherein the candidate list generator module is further to apply at least one token generation rule to the validated token pairs to identify invalid token pairs. 
     
     
         6 . The system of  claim 5 , wherein an invalid token pair is one resulting in a large number of data items in unrelated categories as a result from search. 
     
     
         7 . The system of  claim 2 , wherein the candidate list generator module is further to compare the search query information to the seller-provided descriptor information for data items stored in the database. 
     
     
         8 . The system of  claim 1 , wherein the vocabulary is a stemming and transliteration vocabulary. 
     
     
         9 . A method comprising:
 receiving search query inputs to search a database of data items, the database having a plurality of categories and an associated data dictionary of terms to add to the search query inputs for searching the database;   using at least one processor, identifying demand information that is retrieved from query logs maintained for the search query inputs stored in the database, and supply information comprising seller-provided descriptor information for the data items stored in the database;   generating a set of token pairs based on the search query inputs, each token pair comprising a keyword from at least one search query input and a token, the token being a synonym of the keyword;   applying candidate selection rules to token pairs across the plurality of categories of data items in the database using the demand information and the supply information, generating a set of candidate token pairs;   validating the set of candidate token pairs by calculating a divergence measure for each of the candidate token pairs, wherein the candidate token pairs having divergence values exceeding a divergence threshold are valid token pairs; and   entering the validated token pairs into the data dictionary to form an updated data dictionary; and   providing the validated token pairs in the updated data dictionary in response to a search query containing a keyword corresponding to at least one entry in the updated data dictionary.   
     
     
         10 . The method of  claim 9 , further comprising:
 receiving supply information wherein the supply information is seller-provided descriptor information for the data items stored in the database; and   comparing the search query inputs to the seller-provided descriptor information for data items stored in the database.   
     
     
         11 . The method of  claim 10 , wherein generating a set of token pairs includes generating a set of token pairs resulting in a plurality of token pairs having a same keyword. 
     
     
         12 . A non-transitory machine-readable medium including instructions which, when read by the machine, cause the machine to perform operations comprising:
 receiving search query inputs to search a database of data items, the database having a plurality of categories and an associated data dictionary of terms to add to the search query inputs for searching the database;   identifying demand information that is retrieved from query logs maintained for the search query inputs stored in the database, and supply information comprising seller-provided descriptor information for the data items stored in the database;   generating a set of token pairs based on the search query inputs, each token pair comprising a keyword from at least one search query input and a token, the token being a synonym of the keyword;   applying candidate selection rules to token pairs across the plurality of categories of data items in the database using the demand information and the supply information, generating a set of candidate token pairs;   validating the set of candidate token pairs by calculating a divergence measure for each of the candidate token pairs, wherein the candidate token pairs having divergence values exceeding a divergence threshold are valid token pairs; and   entering the validated token pairs into the data dictionary to form an updated data dictionary; and   providing the validated token pairs in the updated data dictionary in response to a search query containing a keyword corresponding to at least one entry in the updated data dictionary.