System and Method for Indexing Streams Containing Unstructured Text Data

Abstract

A system, method and computer readable medium for indexing streaming data. Data may be received from distributed devices connected via a network. Data elements may be stored and allocated to data blocks and events of the block stores. Non-text data may be converted into a text representation. The data may be split into terms, and term frequencies of each term within each of the event may be calculated. Block-level term frequency statics may be calculated based on the term frequencies. Tree index structures, such as the Y-tree index, may be generated based on the block-level term frequency data. The Y-tree index structures may use the terms as keys and pointers to the corresponding data blocks and block-level term frequency data. A search query may be performed over the tree index structures.

1 - 29 . (canceled) 
     
     
         30 . A method for indexing data, comprising the steps of:
 allocating stored data-elements to data-blocks of block-stores, wherein the stored data-elements are stored in the block-stores, wherein the stored data-elements are allocated via one or more processors to the data-blocks;   further allocating the block-allocated data-elements to events of the data-blocks, wherein each of the data-blocks comprise one or more events, wherein each of the events comprise the block-allocated data-elements of the corresponding data-block, wherein the block-allocated data-elements are allocated via the one or more processors to the events; and,   splitting the event-allocated data-elements into terms, wherein the event-allocated data-elements are split via the one or more processors into the terms, wherein the terms are adapted to be used in tree index structures as keys, wherein the tree index structures are adapted to be generated for the event-allocated data-elements, wherein the tree index structures are generated via the one or more processors.   
     
     
         31 . The method of  claim 30 , further comprising the step of:
 receiving data-streams, wherein the data-streams comprise streamed data-elements, wherein the streamed data-elements are stored via one or more processors in block-stores, wherein the streamed data-elements are said stored data-elements.   
     
     
         32 . The method of  claim 30 , further comprising the step of:
 calculating a term frequencies of each term in each of the events, wherein the term frequencies are calculated via the one or more processors;   calculating block-level term frequency data for the event-allocated data-elements stored in the corresponding data-block based on the term frequencies, wherein the block-level term frequency data is calculated via the one or more processors; and,   generating the tree index structures for the event-allocated data-elements based on the block-level term frequency data, wherein the terms are used in the tree index structures as keys, wherein the tree index structures are calculated via the one or more processors.   
     
     
         33 . A system for indexing data, comprising:
 block-stores adapted to store data-elements;   data-blocks of the block-stores, the stored data-elements being allocated via one or more processors to the data-blocks;   events of the data-blocks, the block-allocated data-elements being further allocated via the one or more processors to the events of the data-blocks, each of the data-blocks comprising one or more events, each of the events comprising the block-allocated data-elements of a corresponding data-block; and,   terms generated via the one or more processors by splitting the event-allocated data-elements, wherein the terms are adapted to be used in tree index structures as keys, wherein the tree index structures are adapted to be generated for the event-allocated data-elements, wherein the tree index structures are generated via the one or more processors.   
     
     
         34 . The system of  claim 33 , wherein the data-elements comprise streamed data-elements, wherein the data-elements are received via data-streams. 
     
     
         35 . The system of  claim 33 , further comprising:
 term frequencies calculated via the one or more processors based on the frequency of each term in each of the event;   block-level term frequency data calculated via the one or more processors for the event-allocated data-elements that are stored in a corresponding data-block, the block-level term frequency data being based on the term frequencies; and,   the tree index structures generated via the one or more processors for the event-allocated data-elements based on the block-level term frequency data, the terms being used in the tree index structures as keys.   
     
     
         36 . A non-transitory computer readable medium having computer readable instructions stored thereon for execution by a processor, wherein the instructions on the non-transitory computer readable medium are adapted to enable a computing device to:
 allocate stored data-elements to data-blocks of block-stores;   further allocate the block-allocated data-elements to events of the data-blocks, wherein each of the data-blocks comprise one or more events, wherein each of the events comprise the block-allocated data-elements of the corresponding data-block; and,   split the event-allocated data-elements into terms, wherein the terms are adapted to be used in tree index structures as keys, wherein the tree index structures are adapted to be generated for the event-allocated data-elements, wherein the tree index structures are generated via the one or more processors.   
     
     
         37 . The non-transitory computer readable medium of  claim 36 , wherein the instructions on the non-transitory computer readable medium are further adapted to enable a computing device to:
 receive data-streams, wherein the data-streams comprise streamed data-elements, wherein the streamed data-elements are stored via one or more processors in block-stores, wherein the streamed data-elements are said stored data-elements.   
     
     
         38 . The non-transitory computer readable medium of  claim 36 , wherein the instructions on the non-transitory computer readable medium are further adapted to enable a computing device to:
 calculate a term frequencies of each term in each of the events, wherein the term frequencies are calculated via the one or more processors;   calculate block-level term frequency data for the event-allocated data-elements stored in the corresponding data-block based on the term frequencies, wherein the block-level term frequency data is calculated via the one or more processors; and,   generate the tree index structures for the event-allocated data-elements based on the block-level term frequency data, wherein the terms are used in the tree index structures as keys, wherein the tree index structures are calculated via the one or more processors.