System and method for multi-dimensional aggregation over large text corpora

Abstract

Systems and methods for multi-dimensional aggregation. Exemplary embodiments include a method for retrieving data from an inverted list index within a computer system, wherein the index comprises annotated postings, the method including receiving a query in a system, converting the query into a query language, scanning at least one list of postings for data from the query, aggregating the data in the list, thereby resulting in an aggregated list, wherein the aggregating includes recording the occurrence of unique values from the list, mapping the values using a user-provided definition to an alternate value, grouping the values by a user-provided mapping of values to groups, recording and mutating data associated with the unique value in the list, relating the recorded data values with other values in the index and returning the requested data from the aggregated list in a return format.

1 . A method for retrieving data from an inverted list index within a computer system, wherein the index comprises annotated postings, the method comprising:
 receiving a query in a system;   converting the query into a query language;   scanning at least one list of postings for data from the query;   aggregating the data in the list, thereby resulting in an aggregated list, wherein the aggregating includes:
 recording the occurrence of unique values from the list; 
 mapping the values using a user-provided definition to an alternate value; 
 grouping the values by a user-provided mapping of values to groups; 
 recording and mutating data associated with the unique value in the list; 
 relating the recorded data values with other values in the index; and 
 returning the requested data from the aggregated list in a return format. 
   
   
   
       2 . The method as claimed in  claim 1  wherein the annotated postings contain per-document identification, per-occurrence identification, and per-occurrence related data, wherein alternately per-occurrence related data is accessible using per-document identification and per-occurrence identification. 
   
   
       3 . The method as claimed in  claim 2  wherein the unique value is the result of a computation on a pre-existing value. 
   
   
       4 . The method as claimed in  claim 3  wherein recording data associated with the unique value takes place during query processing. 
   
   
       5 . A method for multi-dimensional inverted index aggregation within a computer system having an input device, a memory and a display, the method consisting of:
 receiving a search query in the memory from the input device;   converting the query into a query language and sending the request to an index server;   parsing the query and identifying requisite postings lists and aggregation keys and functions;   initializing the aggregation functions;   while results are being collected and prior to the expiration of a pre-determined time, iteratively seeking through the postings list for matches to the query;   passing the aggregation keys to the aggregation functions in response to a match;   processing the keys with a respective function and incrementing a key-specific counter;   entering an index to a table from an output of the functions; and   collecting the aggregation results and returning the results to the display.