Distributed and Interactive Database Architecture for Parallel and Asynchronous Data Processing of Complex Data and for Real-Time Query Processing

Abstract

The various embodiments of the invention provide a data processing system and method, for applications such as marketing campaign management, speech recognition and signal processing. An exemplary system embodiment includes a first data repository adapted to store a plurality of entity and attribute data; a second data repository adapted to store a plurality of entity linkage data; a metadata data repository adapted to store a plurality of metadata modules, with a first metadata module having a plurality of selectable parameters, received through a control interface, and having a plurality of metadata linkages to a first subset of metadata modules; and a multidimensional data structure. The control interface may modify the plurality of selectable parameters in response to received control information. A plurality of processing nodes are adapted to use the plurality of selectable parameters to assemble a first plurality of data from the first and second data repositories and from input data, to reduce the first plurality of data to form a second plurality of data, and to aggregate and dimension the second plurality of data for storage in the multidimensional data structure.

It is claimed: 
     
         1 . A data processing system for marketing campaign management, comprising:
 a plurality of data repositories having a plurality of data structures, one or more data repositories of the plurality of data repositories to store a plurality of entity and attribute data; to store a plurality of entity linkage data comprising a plurality of unique and persistent personal identifiers corresponding to a plurality of people; to store a plurality of comparative confidence levels for matching personal data corresponding to the plurality of unique and persistent personal identifiers, and to store a plurality of metadata modules, one or more of the plurality of metadata modules defining the plurality of data structures, defining a marketing campaign, and further defining a plurality of data processes;   a control interface coupled to the plurality of data repositories, the control interface to receive a plurality of selectable processing parameters;   a data storage system storing a multidimensional data structure; and   a plurality of processing nodes coupled to the plurality of data repositories, to the control interface, and to the data storage system, one or more of the plurality of processing nodes to perform the plurality of data processes, the plurality of data processes comprising assembling a first plurality of data from the one or more data repositories and from input data, reducing the first plurality of data to form a second plurality of data, and dimensioning and aggregating the second plurality of data for storage as the multidimensional data structure in the data storage device.   
     
     
         2 . The system of  claim 1 , wherein the one or more of the plurality of metadata modules further have a plurality of user-modifiable and dynamically selectable data processing parameters determining data process values, data attributes, data confidence levels, and data process ordering. 
     
     
         3 . The system of  claim 2 , wherein the one or more of the plurality of processing nodes further are to perform the plurality of data processes in a selected data process order with one or more selected data process values, data attributes and data confidence levels using the plurality of selectable data processing parameters. 
     
     
         4 . The system of  claim 1 , wherein the one or more of the plurality of processing nodes further are to use the multidimensional data structure to determine a plurality of sets of unique and persistent personal identifiers of the plurality of unique and persistent personal identifiers, and perform a plurality of set operations on the plurality of sets of unique and persistent personal identifiers. 
     
     
         5 . The system of  claim 1 , wherein the one or more of the plurality of metadata modules have a plurality of metadata linkages, the plurality of metadata linkages comprising at least one metadata linkage type selected from the group consisting of: a common or communicated parameter of the plurality of selectable processing parameters, a common or communicated attribute of a plurality of attributes, a common process definition, a common process flow, a communication between processes, and combinations thereof. 
     
     
         6 . The system of  claim 1 , wherein the control interface further comprises a user interface, the user interface to select the plurality of selectable processing parameters, to select input data sources, to select a plurality of attributes for data processing, to select and order a subset of data processes of the plurality of data processes, and to select a plurality of dimensions for aggregation. 
     
     
         7 . The system of  claim 1 , wherein at least one processing node of the plurality of processing nodes is to assemble the first plurality of data by performing at least two data assembly processes selected from the group consisting of: a data input process, a data layout process, a data conversion process, a data correction process, a data audit process, a data matching process, a linkage assignment process, a linkage selection process, a data coding process, a data enhancement process, a data change detection process, a data update process, and combinations thereof. 
     
     
         8 . The system of  claim 1 , wherein at least one processing node of the plurality of processing nodes is to use modeled information to provide a version of the plurality of selectable processing parameters. 
     
     
         9 . The system of  claim 1 , wherein at least one processing node of the plurality of processing nodes is to provide a plurality of aggregation dimensions for the second plurality of data. 
     
     
         10 . The system of  claim 1 , wherein at least one processing node of the plurality of processing nodes is to dimension the second plurality of data using a plurality of attribute ranges selectable through the control interface. 
     
     
         11 . The system of  claim 1 , wherein at least one processing node of the plurality of processing nodes further is to reduce the first plurality of data to form the second plurality of data, and to store in an associated memory interim results comprising entity linkage data having corresponding unique and persistent personal identifiers of the plurality of unique and persistent personal identifiers. 
     
     
         12 . The system of  claim 1 , wherein at least one processing node of the plurality of processing nodes is to reduce the first plurality of data to form the second plurality of data using at least one data reduction process selected from the group consisting of: a data netdown process, a data suppression process, a data prioritization process, and combinations thereof. 
     
     
         13 . The system of  claim 1 , wherein at least one processing node of the plurality of processing nodes is to use the multidimensional data structure to process a query received through the control interface and to provide a query response. 
     
     
         14 . The system of  claim 1 , wherein at least one processing node of the plurality of processing nodes further is to provide a marketing campaign fulfillment process of the plurality of data processes, the marketing campaign fulfillment process comprising at least one process selected from the group consisting of: report generation; postal services; postal presort services; inkjet services; printing preparation services; shipping services; data extraction; email formatting; generation of a magnetic tape, disk or electronic file having selected names and physical addresses; generation of email addresses; selected content determination; selected communication channel determination; variable formatting; variable layout; and combinations thereof. 
     
     
         15 . The system of  claim 1 , wherein the entity linkage data comprising the plurality of unique and persistent personal identifiers links individual people or household entities to a corresponding plurality of attributes, the corresponding plurality of attributes comprising at least two of the attributes selected from the group consisting of: name, current address, name and address variations, previous address, age, gender, income, social security number, vehicle ownership, marital status, spouse name, spouse address, children names, children addresses, household identification, home ownership, purchasing histories, marketing responses, employment histories, and combinations thereof. 
     
     
         16 . A data processing system for marketing campaign management, comprising:
 a plurality of data repositories having a plurality of data structures, one or more data repositories of the plurality of data repositories to store a plurality of entity and attribute data; to store a plurality of entity linkage data comprising a plurality of unique and persistent personal identifiers corresponding to a plurality of people; to store a plurality of comparative confidence levels for matching personal data corresponding to the plurality of unique and persistent personal identifiers, and to store a plurality of metadata modules, one or more of the plurality of metadata modules defining the plurality of data structures, defining a marketing campaign, and further defining a plurality of data processes, and one or more of the plurality of metadata modules further having a plurality of user-modifiable and dynamically selectable data processing parameters determining data process values, data attributes, data confidence levels, and data process ordering;   a control interface coupled to the plurality of data repositories, the control interface to receive the plurality of selectable processing parameters;   a data storage system storing a multidimensional data structure; and   a plurality of processing nodes coupled to the plurality of data repositories, to the control interface, and to the data storage system, one or more of the plurality of processing nodes to perform the plurality of data processes, the plurality of data processes comprising assembling a first plurality of data from the one or more data repositories and from input data, reducing the first plurality of data to form a second plurality of data, and dimensioning and aggregating the second plurality of data for storage as the multidimensional data structure in the data storage device; and one or more of the plurality of processing nodes to use the multidimensional data structure to process a query received through the control interface and to provide a query response.   
     
     
         17 . The system of  claim 16 , wherein the one or more of the plurality of processing nodes further are to perform the plurality of data processes in a selected data process order with one or more selected data process values, data attributes and data confidence levels using the plurality of selectable data processing parameters. 
     
     
         18 . The system of  claim 16 , wherein the one or more of the plurality of processing nodes further are to use the multidimensional data structure to determine a plurality of sets of unique and persistent personal identifiers of the plurality of unique and persistent personal identifiers, and perform a plurality of set operations on the plurality of sets of unique and persistent personal identifiers. 
     
     
         19 . The system of  claim 16 , wherein the one or more of the plurality of metadata modules have a plurality of metadata linkages, the plurality of metadata linkages comprising at least one metadata linkage type selected from the group consisting of: a common or communicated parameter of the plurality of selectable processing parameters, a common or communicated attribute of a plurality of attributes, a common process definition, a common process flow, a communication between processes, and combinations thereof. 
     
     
         20 . The system of  claim 16 , wherein the control interface further comprises a user interface, the user interface to select the plurality of selectable processing parameters, to select input data sources, to select a plurality of attributes for data processing, to select and order a subset of data processes of the plurality of data processes, and to select a plurality of dimensions for aggregation. 
     
     
         21 . The system of  claim 16 , wherein at least one processing node of the plurality of processing nodes is to assemble the first plurality of data by performing at least two data assembly processes selected from the group consisting of: a data input process, a data layout process, a data conversion process, a data correction process, a data audit process, a data matching process, a linkage assignment process, a linkage selection process, a data coding process, a data enhancement process, a data change detection process, a data update process, and combinations thereof. 
     
     
         22 . The system of  claim 16 , wherein at least one processing node of the plurality of processing nodes is to use modeled information to provide a version of the plurality of selectable processing parameters. 
     
     
         23 . The system of  claim 16 , wherein at least one processing node of the plurality of processing nodes is to provide a plurality of aggregation dimensions for the second plurality of data. 
     
     
         24 . The system of  claim 16 , wherein at least one processing node of the plurality of processing nodes is to dimension the second plurality of data using a plurality of attribute ranges selectable through the control interface. 
     
     
         25 . The system of  claim 16 , wherein at least one processing node of the plurality of processing nodes further is to reduce the first plurality of data to form the second plurality of data, and to store in an associated memory interim results comprising entity linkage data having corresponding unique and persistent personal identifiers of the plurality of unique and persistent personal identifiers. 
     
     
         26 . The system of  claim 16 , wherein at least one processing node of the plurality of processing nodes is to reduce the first plurality of data to form the second plurality of data using at least one data reduction process selected from the group consisting of: a data netdown process, a data suppression process, a data prioritization process, and combinations thereof. 
     
     
         27 . The system of  claim 16 , wherein at least one processing node of the plurality of processing nodes further is to provide a marketing campaign fulfillment process of the plurality of data processes, the marketing campaign fulfillment process comprising at least one process selected from the group consisting of: report generation; postal services; postal presort services; inkjet services; printing preparation services; shipping services; data extraction; email formatting; generation of a magnetic tape, disk or electronic file having selected names and physical addresses; generation of email addresses; selected content determination; selected communication channel determination; variable formatting; variable layout; and combinations thereof. 
     
     
         28 . The system of  claim 16 , wherein the entity linkage data comprising the plurality of unique and persistent personal identifiers links individual people or household entities to a corresponding plurality of attributes, the corresponding plurality of attributes comprising at least two of the attributes selected from the group consisting of: name, current address, name and address variations, previous address, age, gender, income, social security number, vehicle ownership, marital status, spouse name, spouse address, children names, children addresses, household identification, home ownership, purchasing histories, marketing responses, employment histories, and combinations thereof. 
     
     
         29 . A computer-implemented data processing method for marketing campaign management, comprising:
 using one or more processors of a plurality of processors, storing, in one or more data repositories of a plurality of data repositories having a plurality of data structures: a plurality of entity and attribute data; a plurality of entity linkage data comprising a plurality of unique and persistent personal identifiers corresponding to a plurality of people; a plurality of comparative confidence levels for matching personal data corresponding to the plurality of unique and persistent personal identifiers; and a plurality of metadata modules, one or more of the plurality of metadata modules defining the plurality of data structures, defining a marketing campaign, and further defining a plurality of data processes;   using a control interface, receiving a plurality of selectable processing parameters;   using one or more processors of the plurality of processors, storing a multidimensional data structure in a data storage system; and   using one or more processors of the plurality of processors, performing the plurality of data processes, the plurality of data processes comprising assembling a first plurality of data from the one or more data repositories and from input data, reducing the first plurality of data to form a second plurality of data, and dimensioning and aggregating the second plurality of data for storage as the multidimensional data structure in the data storage device.   
     
     
         30 . The method of  claim 29 , wherein the one or more of the plurality of metadata modules further have a plurality of user-modifiable and dynamically selectable data processing parameters determining data process values, data attributes, data confidence levels, and data process ordering. 
     
     
         31 . The method of  claim 30 , further comprising:
 using one or more processors of the plurality of processors, performing the plurality of data processes in a selected data process order with one or more selected data process values, data attributes and data confidence levels using the plurality of selectable data processing parameters.   
     
     
         32 . The method of  claim 29 , further comprising:
 using one or more processors of the plurality of processors, using the multidimensional data structure, determining a plurality of sets of unique and persistent personal identifiers of the plurality of unique and persistent personal identifiers, and performing a plurality of set operations on the plurality of sets of unique and persistent personal identifiers.   
     
     
         33 . The method of  claim 29 , wherein the one or more of the plurality of metadata modules have a plurality of metadata linkages, the plurality of metadata linkages comprising at least one metadata linkage type selected from the group consisting of: a common or communicated parameter of the plurality of selectable processing parameters, a common or communicated attribute of a plurality of attributes, a common process definition, a common process flow, a communication between processes, and combinations thereof. 
     
     
         34 . The method of  claim 29 , further comprising:
 using a user interface, selecting the plurality of selectable processing parameters, selecting input data sources, selecting a plurality of attributes for data processing, selecting and ordering a subset of data processes of the plurality of data processes, and selecting a plurality of dimensions for aggregation.   
     
     
         35 . The method of  claim 29 , further comprising:
 using one or more processors of the plurality of processors, assembling the first plurality of data by performing at least two data assembly processes selected from the group consisting of: a data input process, a data layout process, a data conversion process, a data correction process, a data audit process, a data matching process, a linkage assignment process, a linkage selection process, a data coding process, a data enhancement process, a data change detection process, a data update process, and combinations thereof.   
     
     
         36 . The method of  claim 29 , further comprising:
 using one or more processors of the plurality of processors, using modeled information to provide a version of the plurality of selectable processing parameters.   
     
     
         37 . The method of  claim 29 , further comprising:
 using one or more processors of the plurality of processors, providing a plurality of aggregation dimensions for the second plurality of data.   
     
     
         38 . The method of  claim 29 , further comprising:
 using one or more processors of the plurality of processors, dimensioning the second plurality of data using a plurality of attribute ranges selectable through the control interface.   
     
     
         39 . The method of  claim 29 , further comprising:
 using one or more processors of the plurality of processors, reducing the first plurality of data to form the second plurality of data, and storing in an associated memory interim results comprising entity linkage data having corresponding unique and persistent personal identifiers of the plurality of unique and persistent personal identifiers.   
     
     
         40 . The method of  claim 29 , further comprising:
 using one or more processors of the plurality of processors, reducing the first plurality of data to form the second plurality of data using at least one data reduction process selected from the group consisting of: a data netdown process, a data suppression process, a data prioritization process, and combinations thereof.   
     
     
         41 . The method of  claim 29 , further comprising:
 using one or more processors of the plurality of processors, using the multidimensional data structure to process a query received through the control interface and providing a query response.   
     
     
         42 . The method of  claim 29 , further comprising:
 using one or more processors of the plurality of processors, providing a marketing campaign fulfillment process of the plurality of data processes, the marketing campaign fulfillment process comprising at least one process selected from the group consisting of: report generation; postal services; postal presort services; inkjet services; printing preparation services; shipping services; data extraction; email formatting; generation of a magnetic tape, disk or electronic file having selected names and physical addresses; generation of email addresses; selected content determination; selected communication channel determination; variable formatting; variable layout; and combinations thereof.   
     
     
         43 . The method of  claim 29 , wherein the entity linkage data comprising the plurality of unique and persistent personal identifiers links individual people or household entities to a corresponding plurality of attributes, the corresponding plurality of attributes comprising at least two of the attributes selected from the group consisting of: name, current address, name and address variations, previous address, age, gender, income, social security number, vehicle ownership, marital status, spouse name, spouse address, children names, children addresses, household identification, home ownership, purchasing histories, marketing responses, employment histories, and combinations thereof.