Data abstraction system architecture not requiring interoperability between data providers

Abstract

Described are data abstraction systems, methods, and media for aggregating and abstracting data records from data providers, which are not substantially interoperable with each other. Features include data provider connector modules dynamically loaded, based on definitions stored on disk, that facilitate data mapping and individual matching.

What is claimed is: 
     
       1. A data abstraction system comprising:
 a) a data store comprising one or more data records, the data store defined according to a data format schema; 
 b) one or more processors; and 
 c) a computer-readable storage device comprising non-volatile memory, the computer-readable storage device coupled to the one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to provide a horizontally or vertically scalable cloud architecture and perform automated data abstraction operations via the horizontally or vertically scalable cloud architecture, the operations comprising:
 i) maintaining, on the non-volatile memory, a plurality of definitions of data provider connector modules wherein each definition of a data provider connector module comprises a preferred communication protocol; 
 ii) retrieving a selected definition of a data provider connector module from the non-volatile memory without accessing or querying a database implementation; 
 iii) dynamically loading the data provider connector module based on the selected definition, the data provider connector module configured for a data provider endpoint, wherein the data provider endpoint does not interoperate with other data provider endpoints registered with the data abstraction system, the data provider connector module comprising: a sync scheduling definition, an accessor communication protocol client, a data translator module, and a data mapping system; 
 iv) receiving an instruction, in accordance with the sync scheduling definition, to request a data packet comprising one or more new or updated data records from the data provider endpoint over a network, the one or more new or updated data records defined according to a data records schema specified for the data provider endpoint; 
 v) procuring, via the accessor communication protocol client, the data packet from the data provider endpoint over the network, wherein the accessor communication protocol client supports communication protocol switching among a plurality of supported protocols, and wherein the accessor communication protocol client processes the data records based on the preferred communication protocol; 
 vi) processing, by the data translator module, the one or more new or updated data records by performing a set of automated mapping operations to map the data records schema to the data format schema; 
 vii) retrieving, by the data mapping system, one or more of the data records from the data store; 
 viii) assigning weighted values to a specified set of fields and attributes for each of the retrieved data records according to the mapped schemas and based on values for corresponding fields and attributes in the one or more new or updated data records; 
 ix) applying, by the data mapping system, an algorithm to evaluate a record match based on a calculated point value sum of the fields within the data record that match against a calculated match likelihood of N number of the matching fields; 
 x) persisting the one or more new or updated data records to the data store; and 
 xi) making the one or more data records available via a standardized communication protocol or a user interface. 
 
 
     
     
       2. The data abstraction system of  claim 1 , wherein the data provider connector module is dynamically loaded through reflection or introspection of code. 
     
     
       3. The data abstraction system of  claim 1 , wherein the data provider connector module further comprises a test connection method. 
     
     
       4. The data abstraction system of  claim 1 , wherein the accessor communication protocol client connects to the data provider endpoint after authenticating through an authentication service preconfigured at the data provider. 
     
     
       5. The data abstraction system of  claim 4 , wherein the definition of the data provider connector module comprises credentials, a Uniform Resource Identifier (URI), and an authentication protocol for the authentication service. 
     
     
       6. The data abstraction system of  claim 1 , wherein the definition of the data provider connector module further comprises custom automated instructions that define the communication protocol and automated mapping operations for the data provider connector module. 
     
     
       7. The data abstraction system of  claim 1 , wherein the automated mapping operations for the data records schema include custom fields defined by the data mapping system. 
     
     
       8. The data abstraction system of  claim 1 , wherein the automated data abstraction operations further comprise, before persisting the updated data records, encrypting the updated data records. 
     
     
       9. The data abstraction system of  claim 1 , wherein the accessor communication protocol client procures the data packet from the data provider endpoint based on a specified technology standard. 
     
     
       10. The data abstraction system of  claim 1 , wherein the plurality of supported protocols is defined by the definition of the data provider connector module. 
     
     
       11. The data abstraction system of  claim 1 , wherein the instruction is received from a time-based job scheduler, and wherein the instruction comprises a software-based trigger. 
     
     
       12. The data abstraction system of  claim 1 , wherein the automated data abstraction operations further comprise discarding at least one of the data records based on low measures of the weighted values assigned to the specified set of fields and attributes of the at least one of the data records. 
     
     
       13. The data abstraction system of  claim 1 , wherein the data abstraction is defined according to the definition of the data provider connector module. 
     
     
       14. The data abstraction system of  claim 1 , wherein the data store comprises an in-memory database. 
     
     
       15. The data abstraction system of  claim 1 , wherein the data provider connector module is implemented as a node allocated to the data provider endpoint. 
     
     
       16. The data abstraction system of  claim 1 , wherein the data store is load balanced, and wherein the data store is replicated through a master/master replication system. 
     
     
       17. The data abstraction system of  claim 1 , wherein the one or more of the data records are retrieved from the data store based on fuzzy search logic. 
     
     
       18. The data abstraction system of  claim 1 , comprising a plurality of data provider connector modules dynamically loaded based on respective definitions retrieved from the computer-readable storage device, each of the definitions comprising: a sync scheduling definition, an accessor communication protocol client, a data translator module, and a data mapping system. 
     
     
       19. The data abstraction system of  claim 18 , wherein each of the data provider connector modules is hosted at a distinct server, server cluster, or cloud node. 
     
     
       20. The data abstraction system of  claim 1 , further comprising a user device, wherein the user interface is included in a client instantiated on the user device, wherein the user interface is provided at least one of the data records via a standardized communication protocol implemented Application Programming Interface (API), and wherein a last sync time and a device identifier associated with the user device is persisted for the user interface by the API. 
     
     
       21. The data abstraction system of  claim 1 , wherein the one or more data records are made available via a standardized communication protocol coupled with a standardized API. 
     
     
       22. The data abstraction system of  claim 1 , wherein the one or more data records comprises at least one travel record, hotel booking record, motor vehicle record, homeland security record, commercial real estate record, or veterinary record. 
     
     
       23. The data abstraction system of  claim 1 , wherein the data abstraction is defined for the data provider endpoint such that the data provider endpoint does not interoperate with other data provider endpoints registered with the data abstraction system. 
     
     
       24. The data abstraction system of  claim 1 , wherein the non-volatile memory comprises a disk. 
     
     
       25. The data abstraction system of  claim 1 , wherein operation of the system does not require associated software to be installed at the data provider endpoint. 
     
     
       26. A computer-implemented data abstraction method provided via a horizontally or vertically scalable cloud architecture, the method comprising:
 a) maintaining a data store comprising one or more data records, the data store defined according to a data format schema; 
 b) maintaining, on a computer-readable storage device comprising non-volatile memory non-volatile memory, a plurality of definitions of data provider connector modules wherein each definition of a data provider connector module comprises a preferred communication protocol; 
 c) retrieving a selected definition of a data provider connector module from the non-volatile memory without accessing or querying a database implementation; 
 d) dynamically loading the data provider connector module based on the selected definition, the data provider connector module configured for a data provider endpoint, wherein the data provider endpoint does not interoperate with other data provider endpoints registered with the data abstraction system, the data provider connector module comprising: a sync scheduling definition, an accessor communication protocol client, a data translator module, and a data mapping system; 
 e) receiving an instruction, in accordance with the sync scheduling definition, to request a data packet comprising one or more new or updated data records from the data provider endpoint over a network, the one or more new or updated data records defined according to a data records schema specified for the data provider endpoint; 
 f) procuring, via the accessor communication protocol client, the data packet from the data provider endpoint over the network, wherein the accessor communication protocol client supports communication protocol switching among a plurality of supported protocols, and wherein the accessor communication protocol client processes the data records based on the preferred communication protocol; 
 g) processing, by the data translator module, the one or more new or updated data records by performing a set of automated mapping operations to map the data records schema to the data format schema; 
 h) retrieving, by the data mapping system, one or more of the data records from the data store; 
 i) assigning weighted values to a specified set of fields and attributes for each of the retrieved data records according to the mapped schemas and based on values for corresponding fields and attributes in the one or more new or updated data records; 
 j) applying, by the data mapping system, an algorithm to evaluate a record match based on a calculated point value sum of the fields within the data record that match against a calculated match likelihood of N number of the matching fields; 
 k) persisting the one or more new or updated data records to the data store; and 
 l) making the one or more data records available via a standardized communication protocol or a user interface. 
 
     
     
       27. A non-transitory computer-readable storage media encoded with a computer program including instructions executable by one or more processors to provide a horizontally or vertically scalable cloud architecture and perform automated data abstraction operations via the horizontally or vertically scalable cloud architecture, the operations comprising:
 a) maintaining a data store comprising one or more data records, the data store defined according to a data format schema; 
 b) maintaining, on a computer-readable storage device comprising non-volatile memory non-volatile memory, a plurality of definitions of data provider connector modules wherein each definition of a data provider connector module comprises a preferred communication protocol; 
 c) retrieving a selected definition of a data provider connector module from the non-volatile memory without accessing or querying a database implementation; 
 d) dynamically loading the data provider connector module based on the selected definition, the data provider connector module configured for a data provider endpoint, wherein the data provider endpoint does not interoperate with other data provider endpoints registered with the data abstraction system, the data provider connector module comprising: a sync scheduling definition, an accessor communication protocol client, a data translator module, and a data mapping system; 
 e) receiving an instruction, in accordance with the sync scheduling definition, to request a data packet comprising one or more new or updated data records from the data provider endpoint over a network, the one or more new or updated data records defined according to a data records schema specified for the data provider endpoint; 
 f) procuring, via the accessor communication protocol client, the data packet from the data provider endpoint over the network, wherein the accessor communication protocol client supports communication protocol switching among a plurality of supported protocols, and wherein the accessor communication protocol client processes the data records based on the preferred communication protocol; 
 g) processing, by the data translator module, the one or more new or updated data records by performing a set of automated mapping operations to map the data records schema to the data format schema; 
 h) retrieving, by the data mapping system, one or more of the data records from the data store; 
 i) assigning weighted values to a specified set of fields and attributes for each of the retrieved data records according to the mapped schemas and based on values for corresponding fields and attributes in the one or more new or updated data records; 
 j) applying, by the data mapping system, an algorithm to evaluate a record match based on a calculated point value sum of the fields within the data record that match against a calculated match likelihood of N number of the matching fields; 
 k) persisting the one or more new or updated data records to the data store; and 
 l) making the one or more data records available via a standardized communication protocol or a user interface.