System and method for capture of change data from distributed data sources, for use with heterogeneous targets

Abstract

In accordance with an embodiment, described herein is a system and method for capture of change data from a distributed data source system, for example a distributed database or a distributed data stream, and preparation of a canonical format output, for use with one or more heterogeneous targets, for example a database or message queue. The change data capture system can include support for features such as distributed source topology-awareness, initial load, deduplication, and recovery. A technical purpose of the systems and methods described herein includes determination and communication of changes performed to data at a distributed data source that includes a large amount of data across a plurality of nodes, to one or more target computer systems.

What is claimed is: 
     
         1 . A system for capture of change data from a distributed data source, for use with heterogeneous targets, comprising:
 a computer that includes a processor, and a change data capture process manager executing thereon, wherein the change data capture process manager is configured to capture change data from a distributed data source, using a capture process, for use with one or more targets, including:   determining a distributed source topology associated with a plurality of nodes in the distributed data source, wherein each node is associated with a source change trace entity that records data changes that are processed at that node;   accessing the source change trace entities, to determine data changes at the distributed data source, for use with the one or more targets; and   monitoring for a presence of new nodes or unavailability of one or more nodes within the distributed data source, whereupon a source node determined as being unavailable, selecting, from within a plurality of replica nodes at the distributed data source, a replica node from which to obtain change data records.   
     
     
         2 . The system of  claim 1 , wherein the distributed data source is one of a distributed database, or a distributed data stream, or other distributed data source, and wherein the one or more targets include one or more of a database, message queue, or other target. 
     
     
         3 . The system of  claim 1 , wherein the change data capture process manager performs a change data capture process that converts the change data read from the distributed data source, into a canonical format output of the change data, for consumption by the one or more targets. 
     
     
         4 . The system of  claim 3 , whereupon based on a target system to which the change data will be communicated, the canonical format output of the change data is converted to a format used by the target system. 
     
     
         5 . The system of  claim 3 , wherein the change data capture process manager enables support for a new target system to be provided by a pluggable adapter component that reads the canonical format output of the change data and converts it to a format used by the new target system. 
     
     
         6 . The system of  claim 1 , wherein the change data capture process manager performs a deduplication process that provides automatic deduplication of the data provided by the distributed data source. 
     
     
         7 . The system of  claim 1 , wherein the change data capture process manager performs automatic discovery of the distributed source topology associated with the distributed data source system, and provides access to one or more distributed source change trace entity at nodes of the distributed data source system. 
     
     
         8 . The system of  claim 6 , whereupon a change to the distributed source topology associated with the distributed data source system, including one or more nodes being added to or removed from the distributed source topology, the deduplication process detects the change to the distributed source topology. 
     
     
         9 . The system of  claim 1 , whereupon the change data capture process manager determining that a particular node in the distributed data source system, which had been providing records, becomes unavailable, the change data capture process manager performs a recovery process that selects a replica node at which to obtain records. 
     
     
         10 . The system of  claim 1 , wherein if there is more than one replica node with a matching last record, a replica with the maximum record history is selected to feed a partition token found in the last record processed by the unavailable node. 
     
     
         11 . A method for capture of change data from a distributed data source, for use with heterogeneous targets, comprising:
 capturing, by a change data capture process manager, change data from a distributed data source, using a capture process, for use with one or more targets; including   determining a distributed source topology associated with a plurality of nodes in the distributed data source, wherein each node is associated with a source change trace entity that records data changes that are processed at that node;   accessing the source change trace entities, to determine data changes at the distributed data source, for use with the one or more targets; and   monitoring for a presence of new nodes or unavailability of one or more nodes within the distributed data source, whereupon a source node determined as being unavailable, selecting, from within a plurality of replica nodes at the distributed data source, a replica node from which to obtain change data records.   
     
     
         12 . The method of  claim 11 , wherein the distributed data source is one of a distributed database, or a distributed data stream, or other distributed data source, and wherein the one or more targets include one or more of a database, message queue, or other target. 
     
     
         13 . The method of  claim 11 , wherein the change data capture process manager performs a change data capture process that converts the change data read from the distributed data source, into a canonical format output of the change data, for consumption by the one or more targets. 
     
     
         14 . The method of  claim 13 , whereupon based on a target system to which the change data will be communicated, the canonical format output of the change data is converted to a format used by the target system. 
     
     
         15 . The method of  claim 13 , wherein the change data capture process manager enables support for a new target system to be provided by a pluggable adapter component that reads the canonical format output of the change data and converts it to a format used by the new target system. 
     
     
         16 . The method of  claim 11 , wherein the change data capture process manager performs a deduplication process that provides automatic deduplication of the data provided by the distributed data source. 
     
     
         17 . The method of  claim 11 , wherein the change data capture process manager performs automatic discovery of the distributed source topology associated with the distributed data source system, and provides access to one or more distributed source change trace entity at nodes of the distributed data source system. 
     
     
         18 . The method of  claim 16 , whereupon a change to the distributed source topology associated with the distributed data source system, including one or more nodes being added to or removed from the distributed source topology, the deduplication process detects the change to the distributed source topology. 
     
     
         19 . The method of  claim 11 , whereupon the change data capture process manager determining that a particular node in the distributed data source system, which had been providing records, becomes unavailable, the change data capture process manager performs a recovery process that selects a replica node at which to obtain records. 
     
     
         20 . The method of  claim 11 , wherein if there is more than one replica node with a matching last record, a replica with the maximum record history is selected to feed a partition token found in the last record processed by the unavailable node. 
     
     
         21 . A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers cause the one or more computers to perform a method comprising:
 capturing, by a change data capture process manager, change data from a distributed data source, using a capture process, for use with one or more targets; including   determining a distributed source topology associated with a plurality of nodes in the distributed data source, wherein each node is associated with a source change trace entity that records data changes that are processed at that node;   accessing the source change trace entities, to determine data changes at the distributed data source, for use with the one or more targets; and   monitoring for a presence of new nodes or unavailability of one or more nodes within the distributed data source, whereupon a source node determined as being unavailable, selecting, from within a plurality of replica nodes at the distributed data source, a replica node from which to obtain change data records.