Batch materialization using iterative broadcast

Abstract

Systems and methods are described for batch materialization of an incremental change data capture (CDC) changeset. The primary keys are extracted from the incremental CDC changeset and are broadcast to a plurality of executors. By iteratively broadcasting the extracted primary keys in groups, each being a size that is less than a broadcast limitation, the full dataframe of extracted primary keys is broadcast to the executors. Each executor filters a baseline data table based on the extracted primary keys to generate a baseline match dataframe with all primary keys matching the extracted primary keys, and a baseline unmatched dataframe with all primary keys not matching the extracted primary keys. Each executor receives a partitioned incremental CDC changeset and applies the changes to the baseline match dataframe to produce a baseline change dataframe, which is merged with the baseline unmatched dataframe to produce a final changed baseline data table.

What is claimed is: 
     
         1 . A method performed by a computer system configured for batch materialization, the method comprising:
 receiving an incremental change data capture (CDC) changeset comprising a plurality of primary keys associated with corresponding data changes comprising at least one of additions, updates, and deletes;   extracting primary keys from the incremental CDC changeset;   iteratively broadcasting groups of extracted primary keys to each of a plurality of executors;   filtering, by each executor, a baseline data table based on the extracted primary keys to produce a baseline match dataframe and a baseline unmatched dataframe, wherein all primary keys in the baseline match dataframe match the extracted primary keys and wherein all primary keys in the baseline unmatched dataframe do not match the extracted primary keys;   providing a different subset of the incremental CDC changeset to each of the plurality of executors;   applying, by each executor, changes in a received subset of the incremental CDC changeset to the baseline match dataframe to produce a baseline change dataframe; and   merging the baseline change dataframe with the baseline unmatched dataframe to produce a final changed baseline data table and storing the final changed baseline data table in a data lake.   
     
     
         2 . The method of  claim 1 , further comprising representing the extracted primary keys in a trie data structure, wherein the extracted primary keys represented in the trie data structure are iteratively broadcast to each of the plurality of executors. 
     
     
         3 . The method of  claim 1 , further comprising generating the groups of the extracted primary keys, wherein each group of the extracted primary keys has a size that is less than a broadcast limit to the plurality of executors. 
     
     
         4 . The method of  claim 3 , wherein each group of the extracted primary keys has a same size. 
     
     
         5 . The method of  claim 3 , wherein the groups of the extracted primary keys are generated based on congruence of a hash value of each primary key modulo a number of the groups of the extracted primary keys. 
     
     
         6 . The method of  claim 1 , further comprising combining, by each of a plurality of executors, the groups of the extracted primary keys into a full list of extracted primary keys. 
     
     
         7 . The method of  claim 1 , further comprising:
 estimating a memory requirement for the batch materialization; and   determining a number of executors to be used for the batch materialization based on the memory requirement;   wherein the plurality of executors comprises the number of executors.   
     
     
         8 . The method of  claim 1 , further comprising consolidating rows in the incremental CDC changeset before providing the different subset of the incremental CDC changeset to each of the plurality of executors. 
     
     
         9 . The method of  claim 1 , wherein applying, by each executor, the changes in the received subset of the incremental CDC changeset to the baseline match dataframe to produce the baseline change dataframe comprises:
 generating a combined dataset by performing a union with the received subset of the incremental CDC changeset and the baseline match dataframe;   aggregating values associated with each primary key; and   performing at least one of additions, updates, and deletes from the received subset of the incremental CDC changeset to produce the baseline change dataframe.   
     
     
         10 . A computer system configured for batch materialization, comprising:
 one or more processors; and   a memory communicatively coupled with the one or more processors and storing instructions that, when executed by the one or more processors, causes the computer system to:
 receive an incremental change data capture (CDC) changeset comprising a plurality of primary keys associated with corresponding data changes comprising at least one of additions, updates, and deletes; 
 extract primary keys from the incremental CDC changeset; 
 iteratively broadcast groups of extracted primary keys to each of a plurality of executors; 
 filter, by each executor, a baseline data table based on the extracted primary keys to produce a baseline match dataframe and a baseline unmatched dataframe, wherein all primary keys in the baseline match dataframe match the extracted primary keys and wherein all primary keys in the baseline unmatched dataframe do not match the extracted primary keys; 
 provide a different subset of the incremental CDC changeset to each of the plurality of executors; 
 apply, by each executor, changes in a received subset of the incremental CDC changeset to the baseline match dataframe to produce a baseline change dataframe; and 
 merge the baseline change dataframe with the baseline unmatched dataframe to produce a final changed baseline data table and store the final changed baseline data table in a data lake. 
   
     
     
         11 . The computer system of  claim 10 , wherein the computer system is further configured to represent the extracted primary keys in a trie data structure, wherein the extracted primary keys represented in the trie data structure are iteratively broadcast to each of the plurality of executors. 
     
     
         12 . The computer system of  claim 10 , wherein the computer system is further configured to generate the groups of the extracted primary keys, wherein each group of the extracted primary keys has a size that is less than a broadcast limit to the plurality of executors. 
     
     
         13 . The computer system of  claim 12 , wherein each group of the extracted primary keys has a same size. 
     
     
         14 . The computer system of  claim 12 , wherein the groups of the extracted primary keys are generated based on congruence of a hash value of each primary key modulo a number of the groups of the extracted primary keys. 
     
     
         15 . The computer system of  claim 10 , wherein the computer system is further configured to combine, by each of a plurality of executors, the groups of the extracted primary keys into a full list of extracted primary keys. 
     
     
         16 . The computer system of  claim 10 , wherein the computer system is further configured to:
 estimate a memory requirement for the batch materialization; and   determine a number of executors to be used for the batch materialization based on the memory requirement;   wherein the plurality of executors comprises the number of executors.   
     
     
         17 . The computer system of  claim 10 , wherein the computer system is further configured to consolidate rows in the incremental CDC changeset before providing the different subset of the incremental CDC changeset to each of the plurality of executors. 
     
     
         18 . The computer system of  claim 10 , wherein the computer system is configured to apply, by each executor, the changes in the received subset of the incremental CDC changeset to the baseline match dataframe to produce the baseline change dataframe by being configured to:
 generate a combined dataset by performing a union with the received subset of the incremental CDC changeset and the baseline match dataframe;   aggregate values associated with each primary key; and   perform at least one of additions, updates, and deletes from the received subset of the incremental CDC changeset to produce the baseline change dataframe.