Intelligent data search engine

Abstract

A method includes automatically extracting information that matches a predetermined criterion from one or more web pages at one or more web sites and automatically producing one or more extracted data-field names from the information extracted from the one or more web pages at the one or more web sites. The extracted information includes at least one extracted data-field value associated with one of the one or more extracted data-field names. If one of the extracted data-field names matches an existing data-field name in a previously constructed data base including one or more data fields each associated with a data-field name and a data-field value, the method updates an extracted data-field value associated with the data-field name in the data base. If one of the extracted data field names does not match any of the existing data-field names in the data base, the method adds the extracted data-field name to the data base.

1 . A method, comprising:
 automatically extracting, from one or more web pages at one or more web sites, information that matches a predetermined criterion;   automatically producing one or more extracted data-field names from the information extracted from the one or more web pages at the one or more web sites, wherein the extracted information comprises at least one extracted data-field value associated with one of the one or more extracted data-field names;   if one of the extracted data-field names matches an existing data-field name in a previously constructed data base comprising one or more data fields each associated with a data-field name and a data-field value, updating an extracted data-field value associated with the data-field name in the data base; and   if one of the extracted data field names does not match any of the existing data-field names in the data base, adding the extracted data-field name to the data base.   
   
   
       2 . The method of  claim 1 , further comprising:
 constructing the data base comprising a plurality of categories each defined by a category name, wherein each of plurality of categories is associated with a plurality of data fields in the category, wherein each data field is associated with a data-field name and a data-field value.   
   
   
       3 . The method of  claim 2 , wherein one or more data field names in the previously constructed data base are produced using information extracted from a web site. 
   
   
       4 . The method of  claim 1 , wherein the predetermined criterion comprises one or more of a category name or a data-field name in the data base, a company name, a product name, a taxonomy entry, a synonym entry, or a pattern of expressions. 
   
   
       5 . The method of  claim 1 , further comprising:
 if one of the extracted data fields does not match any of the existing data-field names in the data base, writing an extracted data-field value associated with the extracted data-field name to the data base.   
   
   
       6 . The method of  claim 1 , further comprising:
 if one of the extracted data fields does not match any of the existing data-field names in the data base, normalizing extracted data-field value to an extracted data-field name in the data base.   
   
   
       7 . The method of  claim 1 , wherein the extracted information comprises text, alphanumeric characters, or symbols, and the step of automatically producing one or more extracted data-field names comprises conducting morphological analysis on the text, alphanumeric characters, or symbols in the information extracted from the one or more web pages at the one or more web sites. 
   
   
       8 . The method of  claim 7 , wherein the step of conducting morphological analysis comprises one or more of parsing the text, the alphanumeric characters, or the symbols, and matching the text, alphanumeric characters, or symbols with one or more data-field names in the database. 
   
   
       9 . The method of  claim 7 , wherein the step of conducting morphological analysis comprises:
 producing a list of words from the extracted information; and   matching one or more words in the list of words with at least one of the data-field names in the data base.   
   
   
       10 . The method of  claim 7 , wherein the step of conducting morphological analysis comprises:
 assigning weights to the text, alphanumeric characters, or symbols that are in a pre-determined proximity;   identifying the text, alphanumeric characters, or symbols that are in the pre-determined proximity; and   extracting the data-field names from the text, alphanumeric characters, or symbols that are in the pre-determined proximity in accordance with the weight of the text, alphanumeric characters, or symbols in the pre-determined proximity.   
   
   
       11 . The method of  claim 7 , wherein the step of conducting morphological analysis comprises:
 parsing the text, the alphanumeric characters, or the symbols into one or more tokens each including a sequence of the text, the alphanumeric characters, or the symbols;   comparing the tokens parsed from the text, the alphanumeric characters, or the symbols extracted from the information extracted from the one or more web pages, and   producing one or more extracted data-field names in accordance with tokens that are common to the information extracted from the one or more web page at the same web site.   
   
   
       12 . The method of  claim 1 , wherein the information from the one or more web pages at the one or more web sites are expressed in one or more of hypertext markup languages, PDF file, Microsoft Word, javascript, flash, or XML. 
   
   
       13 . The method of  claim 1 , further comprising:
 representing the information extracted from one of the one or more web pages at one of the web sites by a data structure; and   comparing data structures representing different web pages of the web site to produce the one or more extracted data-field names.   
   
   
       14 . The method of  claim 13 , wherein the step of comparing data structure comprises identifying information that is common in data structures representing different web pages at the web site. 
   
   
       15 . The method of  claim 13  wherein the step of comparing data structure comprises producing intersection between the data structures representing different web pages of the web site to produce an extracted data-field name. 
   
   
       16 . The method of  claim 15  wherein the step of producing intersection comprises:
 expressing the information extracted from one of the one or more web pages by a plurality of data vectors each comprising a sequence of symbols;   identifying the common segments of symbols between the data vectors; and   producing an intersection vector in accordance with the identified common segments of symbols.   
   
   
       17 . The method of  claim 16 , wherein the step of identifying the common segments comprises identifying segments of symbols that have the same sequences in the different data vectors. 
   
   
       18 . The method of  claim 13  wherein the data structure comprises one or more data-field names in the data base. 
   
   
       19 . The method of  claim 13 , wherein the data structure comprises one or more of a tree structure, a vector structure, or an object model. 
   
   
       20 . The method of  claim 1 , wherein the step of updating the extracted data field associated with the data-field name in the data base comprises aggregating the data-field values extracted from the web pages at a plurality of data fields in the data base. 
   
   
       21 . The method of  claim 1 , further comprising:
 extracting information from a web page at a web site using the extracted data-field name added to the data base.   
   
   
       22 . A method, comprising:
 automatically extracting, from one or more web pages at one or more web sites, information that matches a predetermined criterion, wherein the predetermined criterion comprises one or more of a category name, a data-field name in a previously constructed data base, a company name, a product name, a taxonomy entry, a synonym entry, or a pattern of expressions, wherein the data base comprises one or more data fields each associated with a data-field name and a data-field value;   automatically producing one or more extracted data-field names from the information extracted from the one or more web pages at the one or more web sites, wherein the extracted information comprises at least one extracted data-field value associated with one of the one or more extracted data-field names;   if one of the extracted data-field names matches an existing data-field name in the data base, updating an extracted data-field value associated with the data-field name in the data base, and   if one of the extracted data field names does not match any of the existing data-field names in the data base, adding the extracted data-field name to the data base and writing an extracted data-field value associated with the extracted data-field name to the data base.   
   
   
       23 . The method of  claim 22 , further comprising:
 automatically constructing the data base comprising a plurality of categories each defined by a category name, wherein each of plurality of categories is associated with a plurality of data fields in the category, wherein each data field is associated with a data-field name and a data-field value.   
   
   
       24 . The method of  claim 22 , further comprising
 if one of the extracted data fields does not match any of the existing data-field names in the data base, normalizing extracted data-field value to an extracted data-field name in the data base before the step of adding.   
   
   
       25 . The method of  claim 24 , wherein the extracted information comprises text, alphanumeric characters, or symbols, and the step of automatically producing one or more extracted data-field names comprises conducting morphological analysis on the text, alphanumeric characters, or symbols in the information extracted from the one or more web pages at the one or more web sites. 
   
   
       26 . The method of  claim 25 , wherein the step of conducting morphological analysis comprises one or more of parsing the text, the alphanumeric characters, or the symbols, and matching the text, alphanumeric characters, or symbols. 
   
   
       27 . The method of  claim 25 , wherein the step of conducting morphological analysis comprises:
 producing a list of words from the extracted information; and   matching one or more words in the list of words with at least one of the one or more data-field names in the data base.   
   
   
       28 . The method of  claim 25 , wherein the step of conducting morphological analysis comprises:
 assigning weights to the text, alphanumeric characters, or symbols that are in a pre-determined proximity;   identifying the text, alphanumeric characters, or symbols that are in the pre-determined proximity; and   extracting the data-field names from the text, alphanumeric characters, or symbols that are in the pre-determined proximity in accordance with the weight of the text, alphanumeric characters, or symbols in the pre-determined proximity.   
   
   
       29 . The method of  claim 22 , wherein the information from the one or more web pages at the one or more web sites are expressed in one or more of hypertext markup language, PDF file, Microsoft Word, Javascript, flash, or XML 
   
   
       30 . The method of  claim 22 , further comprising:
 representing the information extracted from one of the one or more web pages at one of the web sites by a data structure; and   comparing data structures representing different web pages of the web site to produce the one or more extracted data-field names.   
   
   
       31 . The method of  claim 30 , wherein the data structure comprises one or more data-field names in the data base, data-fields, and non-database words or markers, and wherein the step of comparing comprises identifying data-field names, data-fields, and non-database words or markers that are common in data structures representing different web pages at the web site. 
   
   
       32 . The method of  claim 30 , wherein the step of comparing data structure comprises identifying information that is common in data structures representing different web pages at the web site. 
   
   
       33 . The method of  claim 30 , wherein the step of comparing data structure comprises producing intersection between the data structures representing different web pages of the web site to produce an extracted data-field name, data-fields, and non-database words or markers. 
   
   
       34 . The method of  claim 33 , wherein the producing intersection comprises:
 expressing the information extracted from one of the one or more web pages by a plurality of data vectors each comprising a sequence of symbols;   identifying the common segments of symbols between the data vectors; and   producing an intersection vector for the identified common segments of symbols.   
   
   
       35 . The method of  claim 34 , wherein the step of identifying the common segments comprises identifying segments of symbols that have the same sequences in the different data vectors. 
   
   
       36 . The method of  claim 30 , wherein the data structure comprises one or more data-field names in the data base. 
   
   
       37 . The method of  claim 30 , wherein the data structure comprises one or more of a tree structure, a vector structure, or an object model. 
   
   
       38 . The method of  claim 22 , wherein the step of updating the extracted data field associated with the data-field name in the data base comprises aggregating the data-field values extracted from the web pages at a plurality of data fields in the data base. 
   
   
       39 . The method of  claim 22 , further comprising:
 extracting information from a web page at a web site using the extracted data-field name added to the data base.   
   
   
       40 . A method, comprising:
 automatically constructing the data base comprising a plurality of categories each defined by a category name, wherein each of plurality of categories is associated with a plurality of data fields in the category, wherein each data field is associated with a data-field name and a data-field value;   automatically extracting, from one or more web pages at one or more web sites, information that matches a predetermined criterion;   automatically producing one or more extracted data-field names from the information extracted from the one or more web pages at the one or more web sites, wherein the extracted information comprises at least one extracted data-field value associated with one of the one or more extracted data-field names;   if one of the extracted data-field names matches an existing data-field name in the data base, updating an extracted data-field value associated with the data-field name in the data base;   if one of the extracted data field names does not match any of the existing data-field names in the data base, adding the extracted data-field name to the data base and writing an extracted data-field value associated with the extracted data-field name to the data base; and   extracting information from a web page at a web site using the extracted data-field name.