Method and system for storing, organizing and processing data in a relational database

Abstract

A system and method of processing a data query in a data processing system is provided. The data in the data processing system includes a plurality of individual data elements. The data elements are grouped and stored in at least one data unit. The information about the at least one data unit is gathered and stored in at least one information unit. The method comprises receiving the data query to be executed; using the information in the at least one information unit to optimize and execute the query; resolving the data query; and returning results of the data query for use by the data processing system.

1 . A method of resolving a data query in a data processing system, wherein the data in the data processing system includes a plurality of individual data elements arranged in a table having columns and rows, each of the data elements corresponding to a row and column, the table grouped into a plurality of row units, wherein each row unit comprises a plurality of rows of the table, wherein each row unit comprises a different set of rows, and wherein a first data unit corresponds to a first row unit and a first column, wherein the first data unit comprises a plurality of data elements, the plurality of data elements gathered in the first data unit, the first data unit corresponding to a first information unit, the first information unit gathering a first set of information about the first data unit, the first set of information comprising a representation of information that is less than all unique information in the first data unit, comprising:
 executing an execution plan, wherein the execution plan includes a sequence of data processing operations;   using the information in at least the first information unit to identify data units whose data elements are sufficient to complete a first data processing operation;   retrieving a first set of data elements sufficient to complete the first data processing operation;   completing the first data processing operation using the first set of information and the first set of data elements; and   if the first data processing operation is the last operation in resolving the data query, returning the results of the data query for use by the data processing system and if the first data processing operation is not the last operation in resolving the data query, returning intermediate results of the first data processing operation for use by the remaining data processing operations in the execution plan.   
     
     
         2 . The method according to  claim 1 , wherein the rows are organized into the row units according to an ordering of the rows corresponding to the order in which the rows have been received by the data processing system. 
     
     
         3 . The method according to  claim 1 , wherein the rows are organized into the row units in such a way that the data elements in the first data unit are described more accurately by the first information unit than the data elements in a second data unit are described by a respective second information unit. 
     
     
         4 . The method according to  claim 1 , wherein the data elements gathered within the data units are physically stored by the data processing system or by an independent storage system. 
     
     
         5 . The method according to  claim 1 , wherein the data elements of the first data unit are calculated using a calculation plan, the calculation plan being stored in the first information unit, the calculation plan being used only when the first data unit is to be retrieved. 
     
     
         6 . The method according to  claim 1 , wherein the first information unit stores information about more than one data unit. 
     
     
         7 . The method according to  claim 1 , wherein the first information unit stores information about at least one of:
 a value, a subset of values, or a range of values which do not occur in the first data unit;   a value, a subset of values, or a range of values which occur in the first data unit.   
     
     
         8 . The method according to  claim 1 , wherein the first information unit stores information about an occurrence of values that are not typical for the first data unit or multiple data units. 
     
     
         9 . The method according to  claim 1 , wherein the first information unit stores information about at least one of: the number of data elements in the first data unit, the number of non-NULL data elements in the first data unit, the number of NULL data elements in the first data unit, the number of distinct values of data elements in the first data unit, the minimum value out of data elements in the first data unit, the maximum value out of data elements in the first data unit, the total value of numeric data elements in the first data unit, and the total length of alphanumeric data elements in the first data unit. 
     
     
         10 . The method according to  claim 1 , wherein the size of the first information unit is at least an order of magnitude smaller than the size of the first data unit. 
     
     
         11 . The method according to  claim 1 , wherein completing the first data processing operation further comprises:
 retrieving the data elements in a subset of the data units whose data elements are required to be retrieved to complete the first data processing operation to obtain a partial resolution to the first data processing operation; and   using the partial resolution and the first set of information in the first information unit to re-identify whether the first data unit may contain the data elements that are needed to be retrieved in order to complete the operation.   
     
     
         12 . The method according to  claim 11 , wherein the first information unit is used to select one or many data units whose data elements, after their retrieval, result in the most complete partial resolution. 
     
     
         13 . The method according to  claim 11 , wherein the amount of selected data units is at least an order of magnitude lower than the amount of all available data units. 
     
     
         14 . The method according to  claim 11 , wherein retrieving the data elements in at least a portion of the data units and re-identifying any further data units are repeated until the partial resolution is sufficient to complete the operation. 
     
     
         15 . The method according to  claim 1 , wherein the first information unit is used to categorize the first data unit into categories including:
 first data unit that has no data elements relevant for further resolving the data processing operation;   first data unit where all of the data elements in the first data unit are relevant for further resolving the data processing operation; and   first data unit where the information in the respective information unit is insufficient to categorize the first data unit.   
     
     
         16 . The method according to  claim 15 , wherein, if all the data elements in the first data unit are relevant for further resolving the data processing operation, the first set of information stored in the first information unit is used in resolving the data processing operation without retrieving any data elements from the first data unit. 
     
     
         17 . The method according to  claim 15 , further comprising:
 retrieving data elements of a second data unit, wherein information in the respective information units is insufficient to categorize the first data unit and the second data unit;   obtaining a partial resolution to the data processing operation based on the first information unit and the data elements of the second data unit; and   using the partial resolution and the first set of information in the first information unit to re-categorize the first data unit.   
     
     
         18 . The method according to  claim 1 , wherein the first set of information in the first information unit is used to perform a data processing operation on the first data unit and the intermediate results of previous data processing operations in the execution plan. 
     
     
         19 . The method according to  claim 1 , wherein the first set of information in the first information unit is dynamically altered by a result of at least one of the data processing operations in the execution plan. 
     
     
         20 . The method according to  claim 1 , further comprising a second information unit, wherein the second information unit is dynamically created while resolving a data query, the second information unit gathering a second set of information about a subset of data elements of an intermediate result of previous data processing operations in the execution plan. 
     
     
         21 . The method according to  claim 1 , wherein at least one data processing operation is defined using a language of information queries, wherein the information queries are resolved based on information units, with limited access to the data units. 
     
     
         22 . The method according to  claim 21 , wherein the information queries are resolved with no access to the data units. 
     
     
         23 . The method according to  claim 21 , wherein the information queries are resolved by accessing an amount of data units which is at least an order of magnitude lower than the amount of all available data units. 
     
     
         24 . The method according to  claim 21 , wherein the results of the information queries resolved in the data processing operation are stored in a form of new information units which can be used while resolving further data processing operations in the execution plan. 
     
     
         25 . The method according to  claim 1 , wherein the data query is selected from the group consisting of a SQL statement, a statement of SQL language extension and an operation received by a programmable interface. 
     
     
         26 . The method of  claim 1 , wherein the size of the first information unit is not greater than 1% of the size of the first data unit. 
     
     
         27 . The method of  claim 1 , wherein each plurality of row units comprises disjoined sets of rows. 
     
     
         28 . The method of  claim 1 , wherein when the data in the data processing system is arranged in the table it is grouped into row units of a size of 2 16  rows. 
     
     
         29 . The method of  claim 1 , wherein when the data in the data processing system is arranged in the table it is grouped into row units such that the number of rows in a first row unit is different than the number of rows in a second row unit.