Method and system for storing, organizing and processing data in a relational database

Abstract

A method comprises grouping, via a data processing system, a plurality of data elements in a same data type into a plurality of data units. The method further comprises gathering, via the data processing system, information about data elements in a first data unit into a first information unit. The first information unit comprising a representation of information that is less than all unique information in the first data unit. The method also comprises storing, via the data processing system, the first information unit on a computer readable storage device. The method additionally comprises identifying, via the data processing system, based at least in part on the first information unit, whether the data elements in the first data unit are required to be retrieved to resolve a first data query received by the data processing system and to resolve the first data query.

1 . A method comprising:
 grouping, via a data processing system, a plurality of data elements in a same data type into a plurality of data units;   gathering, via the data processing system, information about data elements in a first data unit into a first information unit, the first information unit comprising a representation of information that is less than all unique information in the first data unit;   storing, via the data processing system, the first information unit on a computer readable storage device; and   identifying, via the data processing system, based at least in part on the first information unit, whether the data elements in the first data unit are required to be retrieved to resolve a first data query received by the data processing system and to resolve the first data query.   
     
     
         2 . The method according to  claim 1 , wherein a size of the first information unit does not exceed 1% of a size of the first data unit. 
     
     
         3 . The method according to  claim 1 , wherein the information in the first information unit includes at least one of:
 a minimum value of the data elements in the first data unit, wherein the minimum value is computed according to the data type;   a maximum value of the data elements in the first data unit, wherein the maximum value is computed according to the data type;   a number of non-null values found within the data elements in the first data unit; and   a total value of the data elements in the first data unit, where the total value is computed according to the data type.   
     
     
         4 . The method according to  claim 1 , wherein the first data unit includes numeric data elements and a range of the numeric data elements is divided into a plurality of intervals with indications of whether or not values of the numeric data elements occur within the intervals being stored in the first information unit. 
     
     
         5 . The method according to  claim 4 , wherein there are 1024 or less intervals, not counting a first and a last interval, the first and the last intervals not being stored in the first information unit. 
     
     
         6 . The method according to  claim 4 , wherein the plurality of intervals have non-uniform interval ranges and the non-uniform interval ranges are adjusted to a content of the numeric data elements such that ranges of the intervals with no data occurrences are larger in size than ranges of the intervals having data occurrences. 
     
     
         7 . The method according to  claim 1 , wherein the first data unit includes string data elements and information in the first information unit indicates whether at least one of the data elements in the first data unit contains a given character in a given position. 
     
     
         8 . The method according to  claim 7 , wherein the first information unit includes information about up to a first 64 positions of the string data elements in the first data unit and wherein up to a first 8 positions are not stored if all strings in the first data unit have identical characters at corresponding positions up to the first 8 positions. 
     
     
         9 . The method according to  claim 1 , wherein the information stored in the first information unit corresponds to relationships between the data elements in the first data unit and data elements in a second data unit. 
     
     
         10 . The method according to  claim 9 , wherein the information in the first information unit indicates whether there is any data element in the first data unit having identical value as any data element in the second data unit. 
     
     
         11 . The method according to  claim 1 , wherein the data elements in the first data unit are arranged in a table having columns and rows, each of the data units having a number of data elements from a single column of the table, the data units having data elements corresponding to same rows of the table being grouped into row units. 
     
     
         12 . The method according to  claim 11 , wherein the information stored in the first information unit corresponds to relationships between the data elements in the first data unit and data elements in a second data unit, wherein the first data unit and the second data unit correspond to same rows and are grouped within a first row unit. 
     
     
         13 . The method according to  claim 12 , wherein the first information unit is structured as a directed acyclic graph having nodes, directed edges, a root node, and directed paths from the root node to the nodes, each of the directed edges corresponding to a data condition based on at least one column of data, each of the directed paths corresponding to a conjunction of the data conditions corresponding to the directed edges in the directed path, and wherein a given node that is arrived at by a given directed path includes information about data elements in one of the data units in the first row unit, the corresponding rows of the first row unit satisfying the conjunction of data conditions corresponding to the directed edges in the given directed path. 
     
     
         14 . The method according to  claim 1 , wherein the first data unit contains up to 65,536 data elements. 
     
     
         15 . The method according to  claim 1 , wherein the first information unit is monitored for efficiency in resolving queries, the efficiency of the first information unit measured as inversely proportional to a percentage of queries expected to be received by the data processing system, whose resolving would require retrieving data elements in the first data unit subject to availability of the first information unit, and wherein the first information unit can be deleted because of its low efficiency according to constraints including at least one of:
 a maximum pre-determined amount of disk space used by all of the information units; and   a maximum pre-determined amount of memory used by all of the information units.   
     
     
         16 . The method according to  claim 15 , wherein a second information unit is selected as a candidate to replace the first information unit as associated to the first data unit and made available to the data processing system, the second information unit being selected using heuristic optimization algorithms including at least one of: greedy, randomized, and AI-based, wherein the algorithms comparing the first information unit and the second information unit are based on a group of criteria including at least one of:
 a size of the first information unit versus a size of the second information unit; and   an efficiency of the first information unit versus an efficiency of the second information unit.   
     
     
         17 . The method according to  claim 1 , wherein a new information unit is created for at least one of:
 a part of data which is produced during resolving the first data query received by the data processing system;   a part of data which is a result of resolving the first data query received by the data processing system.   
     
     
         18 . The method according to  claim 17 , wherein at least one of following scenarios is applied to the new information unit after the first data query is resolved by the data processing system:
 the new information unit is not stored after the first data query is resolved;   the new information unit is stored after the first data query is resolved and can be used while resolving a second data query received by the data processing system; and   an intelligent algorithm is applied to decide whether the new information unit should be made available for the data processing system to resolve a second data query after the first data query is resolved.   
     
     
         19 . The method according to  claim 18 , wherein the new information unit is availed for the data processing system to resolve the second data query without storing corresponding data produced during resolving the first data query. 
     
     
         20 . The method according to  claim 1 , wherein a content of the first information unit is changing during resolving the first data query received by the data processing system. 
     
     
         21 . The method according to  claim 20 , wherein at least one of following scenarios is applied:
 changes of the content of the first information unit are not stored after the first data query is resolved;   changes of the content of the first information unit are stored after the first data query is resolved and can be used while resolving a second data query received by the data processing system; and   an intelligent algorithm is applied to decide which of changes of the content of the first information unit should be made available for the data processing system to resolve a second data query after the first data query is resolved.   
     
     
         22 . The method according to  claim 1 , wherein the data elements in the first data unit are stored by at least one of the data processing system and an independent storage system. 
     
     
         23 . The method according to  claim 1 , wherein the data elements in the first data unit are calculated using a calculation plan, the calculation plan being stored as a part of the information in the first information unit, the calculation plan being used if the first data unit is to be retrieved while resolving the first data query received by the data processing system. 
     
     
         24 . The method according to  claim 12 , wherein the rows are organized into row units according to an ordering of the rows corresponding to an order in which the rows have been received by the data processing system. 
     
     
         25 . The method according to  claim 12 , wherein the rows are organized into row units in such a way that the first information unit associated with the first data unit is more efficient than a second information unit associated with a second data unit, wherein the first data unit and the second data unit correspond to same rows and are grouped within a first row unit, and wherein efficiencies of the first information unit and the second information unit are measured as inversely proportional to percentages of queries expected to be received by the data processing system, whose resolving would require retrieving the data elements in the first data unit subject to availability of the first information unit and the second data unit subject to availability of the second information unit.