Supporting scalable distributed secondary index using replication engine for high-performance distributed database systems

Abstract

Implementations of the present disclosure include providing, at each node in a set of nodes of a database system, a table partition of a plurality of table partitions, the plurality of table partitions being provided by partitioning a table using a primary key, providing, at each node in the set of nodes of the database system, a secondary index partition of a plurality of secondary index partitions, each secondary index partition including a replicate table of at least a portion of the table, the plurality of secondary index partitions being provided by partitioning the table using one or more secondary keys, and for at least one operation executed on a table partition, executing a replication protocol to replicate the at least one operation on a secondary index partition that corresponds to the table partition.

What is claimed is: 
     
       1. A computer-implemented method for providing a secondary index within a database system, the method comprising:
 providing, at each node in a set of nodes of the database system, a table partition of a plurality of table partitions, the plurality of table partitions being provided by partitioning a table using a primary key; 
 providing, at each node in the set of nodes of the database system, a secondary index partition of a plurality of secondary index partitions by partitioning a replicate of the table using one or more secondary keys, each secondary index partition being absent a column of a respective primary key and one or more columns of at least one secondary key, each secondary index partition comprising a global index with respect to the one or more secondary keys and referencing each table partition of the plurality of table partitions across all nodes in the set of nodes, each table partition pointing to the secondary index partitions of all nodes in the set of nodes; and 
 for at least one operation executed on a table partition, executing a replication protocol to replicate the at least one operation on a secondary index partition of a node in the set of nodes. 
 
     
     
       2. The method of  claim 1 , wherein a first secondary index partition includes a first set of columns of the table and a second secondary index partition includes a second set of columns of the table, the first set of columns being different from the second set of columns. 
     
     
       3. The method of  claim 1 , wherein the replication protocol comprises an asynchronous replication protocol that is absent a multi-phase commit for committing one or more changes to the table partition that result from the at least one operation to the table partition and the secondary index partition that corresponds to the table partition. 
     
     
       4. The method of  claim 3 , wherein the replication protocol executes an asynchronous replication of the one or more changes prior to committing the one or more changes to the table partition and the secondary index partition that corresponds to the table partition. 
     
     
       5. The method of  claim 1 , wherein the replication protocol comprises a synchronous replication protocol that ensures the one or more changes are represented in the secondary index partition that corresponds to the table partition prior to committing the one or more changes to the table partition and the secondary index partition that corresponds to the table partition. 
     
     
       6. The method of  claim 5 , wherein the replication protocol executes a multi-phase commit for committing the one or more changes to the table partition and the secondary index partition that corresponds to the table partition. 
     
     
       7. The method of  claim 1 , wherein the secondary index partitions of the plurality of secondary index partitions collectively define a global secondary index of the table. 
     
     
       8. A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations for providing a secondary index within a database system, the operations comprising:
 providing, at each node in a set of nodes of the database system, a table partition of a plurality of table partitions, the plurality of table partitions being provided by partitioning a table using a primary key; 
 providing, at each node in the set of nodes of the database system, a secondary index partition of a plurality of secondary index partitions by partitioning a replicate of the table using one or more secondary keys, each secondary index partition being absent a column of a respective primary key and one or more columns of at least one secondary key, each secondary index partition comprising a global index with respect to the one or more secondary keys and referencing each table partition of the plurality of table partitions across all nodes in the set of nodes, each table partition pointing to the secondary index partitions of all nodes in the set of nodes; and 
 for at least one operation executed on a table partition, executing a replication protocol to replicate the at least one operation on a secondary index partition of a node in the set of nodes. 
 
     
     
       9. The computer-readable storage medium of  claim 8 , wherein a first secondary index partition includes a first set of columns of the table and a second secondary index partition includes a second set of columns of the table, the first set of columns being different from the second set of columns. 
     
     
       10. The computer-readable storage medium of  claim 8 , wherein the replication protocol comprises an asynchronous replication protocol that is absent a multi-phase commit for committing one or more changes to the table partition that result from the at least one operation to the table partition and the secondary index partition that corresponds to the table partition. 
     
     
       11. The computer-readable storage medium of  claim 10 , wherein the replication protocol executes an asynchronous replication of the one or more changes prior to committing the one or more changes to the table partition and the secondary index partition that corresponds to the table partition. 
     
     
       12. The computer-readable storage medium of  claim 8 , wherein the replication protocol comprises a synchronous replication protocol that ensures the one or more changes are represented in the secondary index partition that corresponds to the table partition prior to committing the one or more changes to the table partition and the secondary index partition that corresponds to the table partition. 
     
     
       13. The computer-readable storage medium of  claim 12 , wherein the replication protocol executes a multi-phase commit for committing the one or more changes to the table partition and the secondary index partition that corresponds to the table partition. 
     
     
       14. The computer-readable storage medium of  claim 8 , wherein the secondary index partitions of the plurality of secondary index partitions collectively define a global secondary index of the table. 
     
     
       15. A system, comprising:
 one or more computers; and 
 a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations for providing a secondary index within a database system, the operations comprising:
 providing, at each node in a set of nodes of the database system, a table partition of a plurality of table partitions, the plurality of table partitions being provided by partitioning a table using a primary key; 
 providing, at each node in the set of nodes of the database system, a secondary index partition of a plurality of secondary index partitions by partitioning a replicate of the table using one or more secondary keys, each secondary index partition being absent a column of a respective primary key and one or more columns of at least one secondary key, each secondary index partition comprising a global index with respect to the one or more secondary keys and referencing each table partition of the plurality of table partitions across all nodes in the set of nodes, each table partition pointing to the secondary index partitions of all nodes in the set of nodes; and 
 for at least one operation executed on a table partition, executing a replication protocol to replicate the at least one operation on a secondary index partition of a node in the set of nodes. 
 
 
     
     
       16. The system of  claim 15 , wherein a first secondary index partition includes a first set of columns of the table and a second secondary index partition includes a second set of columns of the table, the first set of columns being different from the second set of columns. 
     
     
       17. The system of  claim 15 , wherein the replication protocol comprises an asynchronous replication protocol that is absent a multi-phase commit for committing one or more changes to the table partition that result from the at least one operation to the table partition and the secondary index partition that corresponds to the table partition. 
     
     
       18. The system of  claim 17 , wherein the replication protocol executes an asynchronous replication of the one or more changes prior to committing the one or more changes to the table partition and the secondary index partition that corresponds to the table partition. 
     
     
       19. The system of  claim 15 , wherein the replication protocol comprises a synchronous replication protocol that ensures the one or more changes are represented in the secondary index partition that corresponds to the table partition prior to committing the one or more changes to the table partition and the secondary index partition that corresponds to the table partition. 
     
     
       20. The system of  claim 19 , wherein the replication protocol executes a multi-phase commit for committing the one or more changes to the table partition and the secondary index partition that corresponds to the table partition.