Data partition handoff between storage clusters

Abstract

One example provides a method of migrating a data partition from a first storage cluster to a second storage cluster, the method including determining that the data partition meets a migration criteria for migrating from the first storage cluster to the second storage cluster, on the first storage cluster, preparing partition metadata to be transferred, the partition metadata describing one or more streams within the data partition and one or more extents within each stream, transferring the partition metadata from the first storage cluster to the second storage cluster, directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster, on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only, and on the second storage cluster, performing new ingress for the data partition.

1 . Enacted on a computing system, a method of migrating a data partition from a first storage cluster to a second storage cluster, each storage cluster being implemented via one or more server computers, the method comprising:
 determining that the data partition meets a migration criteria for migrating from the first storage cluster to the second storage cluster;   on the first storage cluster, preparing partition metadata to be transferred, the partition metadata describing one or more streams within the data partition and one or more extents within each stream;   transferring the partition metadata from the first storage cluster to the second storage cluster;   directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster;   on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only; and   on the second storage cluster, performing new ingress for the data partition.   
     
     
         2 . The method of  claim 1 , further comprising, after changing the access attribute of the one or more extents within the data partition to read-only, transferring the one or more extents including underlying data within each extent from the first storage cluster to the second storage cluster. 
     
     
         3 . The method of  claim 1 , wherein determining that the data partition meets the migration criteria comprises determining that the first storage cluster is operating at or near a threshold based upon one or more of transactions per second (TPS), CPU usage, and storage capacity. 
     
     
         4 . The method of  claim 1 , wherein determining that the data partition meets the migration criteria is based upon a decommissioning of the first storage cluster. 
     
     
         5 . The method of  claim 1 , wherein preparing the partition metadata to be transferred comprises:
 quarantining a key range of the data partition;   blocking splits and merges on the key range; and   persisting an intention to begin a handoff of the data partition in a partition table.   
     
     
         6 . The method of  claim 1 , wherein preparing the partition metadata to be transferred comprises, on the first storage cluster, creating mirror streams on the second storage cluster. 
     
     
         7 . The method of  claim 1 , wherein transferring the partition metadata from the first storage cluster to the second storage cluster is performed via an asynchronous API call. 
     
     
         8 . The method of  claim 1 , wherein determining that the data partition meets the migration criteria for migrating from the first storage cluster to the second storage cluster comprises determining that a data storage account comprising the data partition meets the migration criteria for migrating from the first storage cluster to the second storage cluster. 
     
     
         9 . The method of  claim 8 , further comprising updating domain name system (DNS) server information for the data storage account. 
     
     
         10 . The method of  claim 1 , wherein the data partition is a first data partition of a plurality of data partitions determined to meet the migration criteria, the method further comprising:
 on the first storage cluster, preparing second partition metadata to be transferred, the second partition metadata describing one or more streams within a second data partition and one or more extents within each stream of the second data partition;   transferring the second partition metadata from the first storage cluster to the second storage cluster;   directing new transactions associated with the second data partition to the second storage cluster, including while the one or more extents within the second data partition reside at the first storage cluster;   on the first storage cluster, changing an access attribute of the one or more extents within the second data partition to the read-only; and   on the second storage cluster, performing new ingress for the second data partition.   
     
     
         11 . A computing system, comprising:
 a first storage cluster and a second storage cluster, each storage cluster being implemented via one or more server computers; and   memory holding instructions executable by the logic subsystem to:
 determine that a data partition of the first storage cluster meets a migration criteria for migrating the data partition from the first storage cluster to the second storage cluster; 
 on the first storage cluster, prepare partition metadata describing one or more streams within the data partition and one or more extents within each stream; 
 transfer the partition metadata from the first storage cluster to the second storage cluster; 
 direct new transactions associated with the data partition to the second storage cluster, including while the one or more extents remain on the first storage cluster; 
 on the first storage cluster, change an access attribute of the one or more extents within the data partition to read-only; and 
 on the second storage cluster, perform new ingress for the data partition. 
   
     
     
         12 . The computing system of  claim 11 , wherein the instructions are further executable to, after changing the access attribute of the one or more extents within the data partition to read-only, transfer the one or more extents including underlying data within each extent from the first storage cluster to the second storage cluster. 
     
     
         13 . The computing system of  claim 11 , wherein the instructions are executable to determine that the data partition meets the migration criteria by determining that the first storage cluster is operating at or near a threshold based upon one or more of transactions per second (TPS), CPU usage, and storage capacity. 
     
     
         14 . The computing system of  claim 11 , wherein the instructions are executable to determine that the data partition meets the migration criteria based upon a decommissioning of the first storage cluster. 
     
     
         15 . The computing system of  claim 11 , wherein the instructions executable to prepare the partition metadata to be transferred are executable to:
 quarantine a key range of the data partition;   block splits and merges on the key range; and   persist an intention to begin a handoff of the data partition in a partition table.   
     
     
         16 . A method of migrating a data partition from a first storage cluster to a second storage cluster, each storage cluster being implemented via one or more server computers, the method comprising:
 transferring partition metadata from the first storage cluster to the second storage cluster, the partition metadata describing one or more streams within the data partition and one or more extents within each stream;   directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster; and   on the second storage cluster, performing new ingress for the data partition.   
     
     
         17 . The method of  claim 16 , further comprising:
 on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only; and   after changing the access attribute of the one or more extents within the data partition to read-only, transferring the one or more extents including underlying data within each extent from the first storage cluster to the second storage cluster.   
     
     
         18 . The method of  claim 17 , further comprising:
 creating a new extent, responsive to the operation of changing the access attribute, including data directed to be written to the one or more extents associated with the changed access attribute; and   linking the new extent at an end of a stream associated with the one or more extents.   
     
     
         19 . The method of  claim 16 , further comprising:
 adding a redirection instruction to the first storage cluster for directing inquiries from the first storage cluster to the second storage cluster during the data partition migration.   
     
     
         20 . The method of  claim 16 , further comprising:
 determining that the data partition meets a migration criterion for migrating from the first storage cluster to the second storage cluster,   wherein determining that the data partition meets the migration criterion:
 is based on a determination that the first storage cluster is operating at or near a threshold based upon one or more of transactions per second (TPS), CPU usage, and storage capacity; 
 is based on a determination that a data storage account including the data partition meets the migration criterion for migrating from the first storage cluster to the second storage cluster; or 
 is based upon a decommissioning of the first storage cluster. 
   
     
     
         21 . The method of  claim 20 , further comprising updating domain name system (DNS) server information for the data storage account. 
     
     
         22 . The method of  claim 16 , further comprising:
 loading, by the first storage cluster an account table for the second storage cluster; and   loading, by the second storage cluster, an account table for the first storage cluster.   
     
     
         23 . The method of  claim 16 , wherein preparing the partition metadata to be transferred comprises:
 quarantining a key range of the data partition;   blocking splits and merges on the key range; and   persisting an intention to begin a handoff of the data partition in a partition table.   
     
     
         24 . The method of  claim 16 , wherein preparing the partition metadata to be transferred comprises, on the first storage cluster, creating mirror streams on the second storage cluster. 
     
     
         25 . The method of  claim 16 , wherein transferring the partition metadata from the first storage cluster to the second storage cluster is performed via an asynchronous API call. 
     
     
         26 . A computing system, comprising:
 a first storage cluster and a second storage cluster, each storage cluster being implemented via one or more server computers; and   processor-executable instructions stored in memory and executable by a logic subsystem to:
 transfer partition metadata from the first storage cluster to the second storage cluster, the partition metadata describing one or more streams within a data partition and one or more extents within each stream; 
 direct new transactions associated with the data partition to the second storage cluster, including while the one or more extents remain on the first storage cluster; and 
 on the second storage cluster, perform new ingress for the data partition. 
   
     
     
         27 . The computing system of  claim 26 , wherein the processor-executable instructions are further executable to:
 on the first storage cluster, change an access attribute of the one or more extents within the data partition to read-only; and   after changing the access attribute of the one or more extents within the data partition to read-only, transfer the one or more extents including underlying data within each extent from the first storage cluster to the second storage cluster.   
     
     
         28 . The computing system of  claim 27 , wherein the processor-executable instructions are further executable to:
 create a new extent, responsive to the change of the access attribute, including data directed to be written to the one or more extents associated with the changed read-only access attribute;   and link the new extent at an end of a stream associated with the one or more extents.   
     
     
         29 . The computing system of  claim 26 , wherein the processor-executable instructions are further executable to:
 add a redirection instruction to the first storage cluster for directing inquiries from the first storage cluster to the second storage cluster responsive to determining that the data partition satisfies a migration criterion for migrating from the first storage cluster to the second storage cluster.   
     
     
         30 . The computing system of  claim 26 , wherein the processor-executable instructions are further executable to:
 determine that a data partition of the first storage cluster meets a migration criterion for migrating the data partition from the first storage cluster to the second storage cluster,   wherein the instructions are executable to determine that the data partition meets the migration criterion:
 by determining that the first storage cluster is operating at or near a threshold based upon one or more of transactions per second (TPS), CPU usage, and storage capacity; 
 by determining that a data storage account comprising the data partition meets the migration criterion for migrating from the first storage cluster to the second storage cluster; or 
 based upon a decommissioning of the first storage cluster. 
   
     
     
         31 . The computing system of  claim 26 , wherein the processor-executable instructions are further executable to:
 quarantine a key range of the data partition;   block splits and merges on the key range; and   persist an intention to begin a handoff of the data partition in a partition table.   
     
     
         32 . The computing system of  claim 26 , wherein the processor-executable instructions are further executable to create mirror streams of the first storage cluster on the second storage cluster. 
     
     
         33 . The computing system of  claim 26 , wherein the processor-executable instructions are further executable to transfer the partition metadata from the first storage cluster to the second storage cluster via an asynchronous API call. 
     
     
         34 . The computing system of  claim 26 , wherein the data partition is a first data partition of a plurality of data partitions determined to meet a migration criterion, and wherein the processor-executable instructions are further executable to:
 transfer second partition metadata from the first storage cluster to the second storage cluster, the second partition metadata describing one or more streams within a second data partition and one or more extents within each stream of the second data partition;   direct new transactions associated with the second data partition to the second storage cluster, including while the one or more extents within the second data partition reside at the first storage cluster;   on the first storage cluster, change an access attribute of the one or more extents within the second data partition to be read-only; and   on the second storage cluster, perform new ingress for the second data partition.   
     
     
         35 . One or more tangible processor-readable storage media embodied with instructions for executing on one or more processors and circuits a process for migrating a data storage account from a first storage cluster to a second storage cluster, the process comprising:
 for each key range of a plurality of key ranges within the data storage account, transferring metadata for the key range from the first storage cluster to the second storage cluster;   once metadata for all key ranges within the data storage account has been transferred from the first storage cluster to the second storage cluster, updating domain name system (DNS) server information for the data storage account; and   receiving customer requests at the second storage cluster, including when a customer request involves data not yet migrated from the first storage cluster to the second storage cluster.