Method and system for transforming input data streams

Abstract

The present system and method transforms an input data stream in a first data format of a plurality of first data formats to an output data stream in a second data format of a plurality of second data formats. A plurality of input connector modules receive respective input data streams and at least one input queue stores the received input data streams. A plurality of job threads is operatively connected to the at least one input queue, each job thread, in parallel with at least one other job thread, formatting a stored input data stream to produce an output data stream. At least one output queue respectively stores the output data streams from the plurality of job threads. A plurality of output connector modules is operatively connected to the at least one output queue, the output connector modules supplying respective output data streams.

1 . A system for transforming an input data stream in a first data format of a plurality of first data formats to an output data stream in a second data format of a plurality of second data formats, comprising:
 a parsing model comprising information about message structure; at least one input queue for storing input data streams;   a plurality of job threads operatively connected to the at least one input queue, each job thread, in parallel with at least one other job thread, formatting a respective stored input data stream to produce an output data stream, each job thread comprising:
 a filter for filtering irrelevant data from the respective retrieved input data stream; 
 an event identifier for identifying events in the input data stream, the events comprising sequences, patterns or both, in the input data stream: a field list for each of the identified events the fields comprising data, variables, or both in the input stream; 
 a message generator for generating messages associated with each identified event in response to the field list for the identified event and the parsing model; and 
 a processor for processing each message in response to the parsing model to generate meta records, the meta records comprising output data that is not formatted for a specific output device.