Method and apparatus for storing data as objects, constructing customized data retrieval and data processing requests, and performing householding queries

Abstract

Compressed data objects are stored in a remote data store and are accessible to end users for data retrieval and processing. The remote data store is optimized for efficient data storage and retrieval with processes for capturing data profile statistics and creating an index of compressed data. Descriptive information about data objects and processing logic is displayed and made available for viewing and access by end users having a client application which manipulates the descriptive information to create a customized data retrieval and processing request. An end user can add descriptive information about its own data to the request and attach its own compressed data to the request that will be included in data retrieval and processing. Descriptive information representing data and logic stored in multiple remote data stores can be integrated into one request. The request can be submitted to several data stores in succession for automated data retrieval and processing without jeopardizing request security. A final, customized request object, thus, may contain descriptive information identifying data, descriptive information identifying logic to be executed on the data, and compressed end user data. The final request is submitted to the appropriate data store(s) and/or vendor(s) for automated processing in accordance with the customized instructions, and the data result set is returned in a format specified by the end user. The contents of the object data store and the request can be processed substantially in compressed form. Householding capability and security features are available. End users can customize their householding parameters, including end user specifications to improve flexibility, speed, and efficiency of the householding process.

What is claimed is:  
     
         1 . A method of storing data in a data store for subsequent search and retrieval by an end user, comprising: 
 receiving input data to be stored for subsequent search and retrieval;    processing the received input data by transforming the input data into an indexed and compressed universal data object; and    storing the indexed and compressed universal data object in a data store for subsequent search and retrieval by an end user.    
     
     
         2 . The method of storing data according to  claim 1 , wherein the transforming includes 
 generating data profile statistics;    compressing the data; and    indexing the compressed data.    
     
     
         3 . A method of searching data in the data store according to the method of  claim 1 , from an end user local system having a graphical user interface for interactively generating an application programming language search request, the searching comprising: 
 extracting and displaying information about the data stored at the data store, and making data processing logic available at the data store;    accessing the information with the end user graphical user interface, displaying this information on the end user graphical user interface, and interactively generating an executable request for data using the application programming language; and    providing the executable request to the data store for processing to produce search results.    
     
     
         4 . The method of searching data according to  claim 3 , wherein the end user local system has a local end user data store containing data stored according to the method of  claim 1 , 
 wherein the displaying includes displaying with the end user graphical user interface, the end user local data, associated data profile statistics and data processing logic; and    wherein the executable request includes: 
 data object header of the end user data; and  
 an attachment including compressed end user data.  
   
     
     
         5 . A method of retrieving data from a data store, using the searching method according to  claim 3 , comprising: 
 executing the executable request at the data store having an indexed and compressed universal data object, in accordance with the corresponding application programming language, substantially without decompressing the data; and    forwarding the results of the executing to the end user, the results remaining in compressed form.    
     
     
         6 . The method according to  claim 3  wherein an end user can perform a householding request.  
     
     
         7 . The method of storing according to  claim 1 , wherein the data to be processed is stored in files of any type containing records, and wherein the processing includes creating a data store of compressed universal data objects, the creating comprising: 
 selecting data files to be processed;    entering information to be used for a data object header identifying and describing associated input data, the information including a source identification which identifies by source the type of records contained in a file, a key field for each file, used to uniquely identify a record or group of records, and a record layout representing records stored in a file, such as the type and length of data fields;    scanning the data and associated data definitions;    mapping the data to the corresponding data object header;    generating data profile statistics; and    compressing the data object header information and input data.    
     
     
         8 . The method of storing according to  claim 7 , wherein the key field is at least one record field or a combination of the source identification and a particular data field or other information selected by the end user.  
     
     
         9 . The method of storing according to  claim 7 , wherein the data object header may further include the date a record was created, the location of file creation, a file owner, and other information about the raw data contained in the associated universal data object.  
     
     
         10 . The method according to  claim 7  wherein the data store extracts the data object header information from the universal data object, comprising: 
 targeting the data object header by referencing to a data object identifier;  
 forming a separate and distinct record containing the data object header information; and  
 transferring the data object header using TCP/IP to a central storage area maintained on a network server;  
 wherein the end user can view the data object header information in order to survey the type of data input contained in the data object store, including the data profile statistics.  
 
     
     
         11 . A database comprising compressed universal data objects stored in a data store and accessible to end users for data retrieval and processing; 
 wherein the data store is optimized for efficient data storage and retrieval with processes for generating data profile statistics and automatically creating an index of compressed data;    wherein descriptive information about universal data objects and processing logic is displayed and made available for viewing and access by end users having access to a client application;    wherein the client application manipulates the descriptive information to create a customized data retrieval and processing request;    wherein an end user can add descriptive information about its own data to the request and attach its own compressed data to the request so that it will be included in data retrieval and processing;    wherein descriptive information representing data and logic stored in multiple remote data stores can be integrated into one request;    wherein the request can be submitted to several data stores in succession for automated data retrieval and processing without jeopardizing request security;    wherein a final, customized request object may contain:    (i) descriptive information identifying data;    (ii) descriptive information identifying logic to be executed on the data; and    (iii) compressed end user data;    wherein the final request is submitted to the appropriate data store(s) and/or vendor(s) for automated processing in accordance with the customized instructions, and the data result set is returned in a format specified by the end user;    wherein the contents of the object data store and the request can be processed substantially entirely in compressed form;    wherein a householding process capability and security features may be included;    wherein end users can customize householding parameters, including end-user specifications to improve the speed and efficiency of the householding process.    
     
     
         12 . The database according to  claim 11 , wherein the security features comprise 
 dynamically compressing and dispersing an identification key needed to open an universal data object;    permitting limited access to information until the request is assembled and the identification key is located; and    limiting information accessed at each data store, wherein the flexibility to circulate a request to several remote data stores consecutively is preserved.    
     
     
         13 . The database according to  claim 11  wherein the data at the data store is organized into data blocks by the number of bytes of the record.  
     
     
         14 . The database according to  claim 11  wherein the data store is accessed by a standard Internet TCP/IP addressing convention.  
     
     
         15 . A method of performing householding queries by comparing input data with reference lists to produce data output, comprising: 
 building input patterns from data input submitted by an end user;    displaying the input patterns for viewing by an end user utilizing a graphical user interface;    allowing the end user to construct an output table containing output table symbols;    mapping the input patterns to the output table symbols;    mapping output table symbols to a reference list;    generating search patterns derived from the mapping of input patterns to the output table symbols that are mapped to the reference list; and    using the search patterns to parse the data input against the reference lists to produce data output.    
     
     
         16 . The method according to  claim 15  wherein the reference lists are compared with the data input, comprising: 
 customizing the input and output layout;  
 building output table symbols corresponding to the data fields contained in the reference list(s);  
 identifying those fields in the reference lists that are relevant for parsing;  
 selecting what constitutes a match and error tolerance fore the search;  
 examining the reference list to determine the number of bytes contained in each field;  
 constructing at least one parsing table created from the reference lists using only those fields that are necessary to find the matches requested by the end user on the graphical user interface and sorting the parsing table(s) into blocks according to the number of bytes; and  
 building a series of execution tables designed to optimize the parsing search.  
 
     
     
         17 . The method of building the parsing table(s) claimed in  16  wherein the invention uses the end user search pattern information to construct indices on the parsing table.  
     
     
         18 . The method of using execution tables claimed in  16 , wherein the first execution table performs a structural search comprising: 
 analyzing the search pattern generate for a given input pattern and identifies the output table symbol for which an exact match is required most frequently in the same search pattern position;    prioritizing the search patterns to eliminate the largest number of non matching records;    selecting key field(s) from the end user's search pattern that will be used in the search process that will eliminate the maximum number of search patterns that will not match the parsing table;    matching data from the parsing table index that matches the search pattern of the key field(s) parameter;    moving the matched data into a virtual memory table so that the successive execution table can retrieve data for next key field(s) without losing the location of the matched data for the previous key field(s); and    storing the frequency of matches to the key field(s) search in a virtual memory table.    
     
     
         19 . The method according to  claim 15  wherein the end user uses the graphical user interface to construct the output table symbols and map a single output table symbol to multiple reference lists.