Systems, methods, and software for distributed loading of databases

Abstract

The present inventors have devised one or more systems, methods, and software for distributed loading of information retrieval systems. One exemplary system includes two or more (at least two) load monitor servers that not only monitor and ensure completion of load tasks by individual load servers in a set of two or more load servers, but also provide for one load monitor to monitor performance of the another. Moreover, the exemplary system provides a service-level-agreement (SLA) data structure for each load server. The SLA data structure governs what types and priority levels of loading tasks will be performed for predetermined time periods.

1 . An information retrieval system comprising:
 a queue for storing load data requests;   a set of two or more load servers for retrieving data from a queue and indexing documents for retrieval based on the data retrieved from the queue;   first and second load monitors for monitoring operation of the load servers and providing new data on the queue accessible by the set of loaders, with the first load monitor configured to provide information regarding its operations to the second load monitor, and the second load monitor configured to respond to status information indicating failure of the first load monitor to take over monitoring of the load servers and providing new data on the queue.   
   
   
       2 . The system of  claim 1 , wherein to provide status information to the second load monitor, the first load monitor is configured to periodically write a time-stamped report to a predetermined portion of a network accessible storage device and the second load monitor is configured to read the time-stamped report on the network accessible storage device at predetermined intervals. 
   
   
       3 . The system of  claim 1 , wherein the first load monitor is configured to notify each of the load servers when new data is added to the queue, and wherein any of the load servers that is not busy is configured to respond to the notification by attempting to retrieve the new data from the queue and to index the documents based on the new data. 
   
   
       4 . The system of  claim 1 , further comprising a set of two or more distribution servers configured to respectively distribute index data generated by the load servers and corresponding documents to a primary index and a primary database for use by subscribers. 
   
   
       5 . The system of  claim 4 :
 wherein each load server is configured to provide a loader notification to each of the distribution servers to indicate that it has completed indexing a set of documents based on data retrieved from the queue; and   wherein any of the distribution servers that is not busy is configured to respond to the loader notification by attempting to distribute index data generated by the load server that provided the loader notification to the master index and to distribute documents corresponding to the loader notification to the database.   
   
   
       6 . The system of  claim 5 , wherein each distribution server is configured to provide a distribution notification to each of the load monitors to indicate that it has completed distributing a set of documents based on data retrieved from the queue to the master index and to the database. 
   
   
       7 . The system of  claim 6 , further comprising means, responsive to the distribution notification, for replicating the distributed index data and documents to a secondary index and database. 
   
   
       8 . The system of  claim 5 , wherein each load server is configured to check the queue for new data after providing the loader notification. 
   
   
       9 . The system of  claim 7 , wherein the first load monitor is configured to confirm replication of the distributed index data and documents in the secondary index and database and in response to confirming replication writing a time-stamped load completion message to a network accessible storage device; and wherein the second load monitor is configured to periodically determine whether the network accessible storage device includes messages confirming proper operation of the first load monitor. 
   
   
       10 . The system of  claim 1 , wherein each load server is associated with a service-level-agreement data structure, the data structure including a first field for identifying for which databases the load server is permitted to index documents; a second field for indicating a minimal priority of document indexing requests that the load server is permitted to accept; and a third field indicating a temporal aspect. 
   
   
       11 . An information retrieval system comprising:
 first and second load monitors for monitoring operation of a plurality of load servers that index documents for addition to a database,   wherein the first load monitor is configured to provide status information regarding its operation to the second load monitor, and the second load monitor is configured to respond to status information indicating failure of the first load monitor to take over monitoring of the load servers.   
   
   
       12 . The system of  claim 11 , wherein to provide status information to the second load monitor, the first load monitor included means for periodically writing a time-stamped report to a predetermined portion of a network accessible storage device and the second load monitor includes means for reading the time-stamped report on the network accessible storage device at predetermined intervals. 
   
   
       13 . The system of  claim 12 , wherein the first load monitor is configured to notify each of the load servers when new documents are ready to be added to the database, and wherein any of the load servers that is not busy is configured to respond to the notification by attempting to retrieve the new data from the queue and to index the documents based on the new data. 
   
   
       14 . A method comprising:
 receiving a file containing documents for addition to a database of an online information-retrieval system;   notifying two or more load servers in response to receiving the file;   indexing the documents in the file using one of the notified load servers;   loading the indexed documents onto the database;   notifying two or more load monitors that the documents have been loaded on the database;   confirming that the documents have been loaded on the database using a first one of the notified load monitors; and   confirming proper operation of a first one of the notified load monitors using a second one of the notified load monitors.   
   
   
       15 . The method of  claim 14 , wherein notifying two or more load servers, comprises:
 detecting reception of the file;   adding data regarding the file to a load queue; and   notifying the two or more load servers that new data has been added to the load queue.   
   
   
       16 . The method of  claim 14 , wherein loading the indexed documents onto the database, comprises:
 notifying two or more of distribution servers that index data and corresponding documents are available for loading; and   using one of the two or more distribution servers to load the index data and corresponding documents on to the database.   
   
   
       17 . The method of  claim 14 , wherein confirming that the documents have been loaded on the database includes reading an “end of file” or other analogous indicator from the database. 
   
   
       18 . The method of  claim 15 , wherein confirming proper operation of a first one of the load monitors using a second one of the load monitors, comprises determining whether the first one of the load monitors has written one or more messages to a shared directory. 
   
   
       19 . The method of  claim 15 , wherein indexing the documents in the file using one of the notified load servers, includes reviewing a service-level-agreement data structure to determine whether one or more of the notified load servers are authorized to process the documents. 
   
   
       20 . A service-related data structure for use in an information retrieval system having a plurality of load servers, the data structure associated with one of the plurality of load servers and including means for indicating a priority level of load requests that the one load server is permitted to process. 
   
   
       21 . The data structure of  claim 20 , further including means for indicating which of a set of databases the load server may process load requests. 
   
   
       22 . The data structure of  claim 20 , further including means for indicating a time period associated with one or more service restrictions for the load server.