Multi-tier cloud system

Abstract

The technology disclosed relates to a STRADL file system that transparently deploys file blocks across multiple tiers of storage. In a cloud-based storage platform, multiple storage tiers host data via file system APIs. This includes a volatile storage (VS) tier with a VS API, a reliable non-volatile storage (RNVS) tier with a RNVS API, and a highly reliable non-volatile storage (HRNVS) tier with a HRNVS API. The STRADL file system provides an intermediary file system API that presents to a host system a single interface and translates get, put, and other file system requests into commands compatible with the VS API, the RNVS API, and the HRNVS API, without host system awareness of which of the multiple tiers holds requested data and metadata.

What we claim is: 
     
         1 . A file system running on a node that transparently deploys file blocks across multiple tiers of storage, with flushing and synchronizing of data from volatile to reliable and highly reliable non-volatile storage, the system comprising:
 multiple tiers of storage that host data via file system application programming interfaces (abbreviated APIs), including:
 volatile storage (abbreviated VS) tier with a VS API; 
 reliable non-volatile storage (abbreviated RNVS) tier with a RNVS API; and 
 highly reliable non-volatile storage (abbreviated HRNVS) tier with a HRNVS API; 
   an intermediary file system API that presents to a host system a single interface and translates file system requests received via different access protocols into commands compatible with the VS API, the RNVS API, and the HRNVS API, without host system awareness of which of the multiple tiers holds requested data and metadata;   a write manager that writes data, received via the intermediary file system API and destined for a file, to the volatile storage tier and marks it to be committed to the reliable non-volatile storage tier;   a consistency point flush manager that periodically commits data from the volatile storage tier to the reliable non-volatile storage tier; and   a synchronization manager that periodically synchronizes data from the reliable non-volatile storage tier to the highly reliable non-volatile storage tier.   
     
     
         2 . The system of  claim 1 , wherein the intermediary file system API controls storage of data across the storage tiers and interaction with the stored data based at least on a cost optimization policy and/or storage task selected by an organization. 
     
     
         3 . The system of  claim 2 , wherein the cost optimization policy and/or the storage task maps to service level objectives (abbreviated SLOs), including at least budget SLOs, cost SLOs, performance SLOs, health SLOs, data protection SLOs, and cloning SLOs. 
     
     
         4 . The system of  claim 3 , wherein the intermediary file system API automatically determines storage parameters that meet the SLOs based at least on:
 cost metrics of the storage tiers, including storage cost, transmission cost, and access cost;   performance characteristics of the storage tiers;   durability characteristics of the storage tiers; and   efficiency characteristics of the storage tiers.   
     
     
         5 . The system of  claim 1 , wherein the different access protocols further include at least network file system (abbreviated NFS), common internet file system (abbreviated CIFS), representational state transfer (abbreviated REST), internet small computer systems interface (abbreviated iSCSI), server message block (abbreviated SMB), file transfer protocol (abbreviated FTP), cloud data management interface (abbreviated CDMI), and apple filing protocol (abbreviated AFP). 
     
     
         6 . The system of  claim 1 , wherein the reliable non-volatile storage tier is mirrored, and the consistency point flush manager periodically mirrors data from the volatile storage tier to an intermediately reliable non-volatile storage, and
 wherein the synchronization manager periodically synchronizes data from the intermediately reliable non-volatile storage to the highly reliable non-volatile storage tier.   
     
     
         7 . The system of  claim 6 , wherein the synchronization manager demirrors redundant persistence of data blocks stored in the intermediately reliable non-volatile storage following synchronization of the data blocks to the highly reliable non-volatile storage, and marks the synchronized data in the reliable non-volatile storage tier as a read cache instead of a write cache. 
     
     
         8 . The system of  claim 1 , wherein the highly reliable non-volatile storage tier hosts a third native file system, the third native file system has third characteristics, and the highly reliable non-volatile storage tier is slower and less expensive than the reliable non-volatile storage tier. 
     
     
         9 . The system of  claim 8 , wherein the reliable non-volatile storage tier hosts a second native file system, and the second native file system has second characteristics that are disjoint from the third characteristics of the third native file system. 
     
     
         10 . The system of  claim 9 , wherein the volatile storage tier hosts a first native file system, and the volatile storage tier is faster and more expensive than the reliable non-volatile storage tier. 
     
     
         11 . The system of  claim 1 , wherein the multiple tiers of storage include an instance non-volatile storage tier that hosts a fifth native file system, and the instance non-volatile storage tier is faster and less reliable than the reliable non-volatile storage tier and is slower and more reliable than the volatile storage tier. 
     
     
         12 . The system of  claim 1 , wherein the write manager includes one or more processors that perform initial writing operations including:
 receiving, from a client, a new write request to write data;   sending the data to the volatile storage tier and in parallel to the transaction log;   receiving, from the volatile storage tier and the transaction log, a write completion message; and   providing an acknowledgment, to the client, acknowledging that the initial writing operations are successful.   
     
     
         13 . The system of  claim 1 , wherein the consistency point flush manager includes one or more processors that perform commit operations including:
 temporarily freezing data in the volatile storage tier at consistency points; and   copying data that has changed between consistency points in the volatile storage tier to the reliable non-volatile storage tier.   
     
     
         14 . The system of  claim 1 , wherein the synchronization manager includes one or more processors that perform synchronization operations including:
 freezing data in the reliable non-volatile storage tier during durable snapshots; and   copying changed durable snapshots from the reliable non-volatile storage tier to the highly reliable non-volatile storage tier.   
     
     
         15 . The system of  claim 1 , wherein the volatile storage tier includes a cache manager that:
 tracks staleness of each data block that already has been copied to the reliable non-volatile storage tier; and   when a data block staleness has exceeded a criteria, expires the stale data block from the volatile storage tier and updates a block table to indicate that the stale data block is to be retrieved from the reliable non-volatile storage tier.   
     
     
         16 . The system of  claim 15 , wherein the reliable non-volatile storage tier includes a cache manager that:
 tracks staleness of each data block that already has been copied to the highly reliable non-volatile storage tier; and   when a data block staleness has exceeded a criteria, expires the stale data block from the reliable non-volatile storage tier and updates a block table to indicate that the stale data block is to be retrieved from the highly reliable non-volatile storage tier.   
     
     
         17 . The system of  claim 16 , further configured to include a read manager with one or more processors that perform read operations via the intermediary file system API irrespective of a storage tier from which data is fetched in response to a read request, the read manager performing read operations including:
 determining which storage tier is a fastest storage tier that holds the data responsive to the read request; and   retrieving the data responsive to the read request, and when the fastest storage tier that holds the data is not the volatile storage tier, caching the retrieved data in the volatile storage tier.   
     
     
         18 . The system of  claim 1 , wherein the multiple tiers of storage are distributed across different cloud-based storage platforms. 
     
     
         19 . A method of transparently deploying file blocks across multiple tiers of storage, with flushing and synchronizing of data from volatile to reliable and highly reliable non-volatile storage, the method including:
 hosting data across multiple tiers of storage via file system application programming interfaces (abbreviated APIs), including:
 volatile storage (abbreviated VS) tier with a VS API, 
 reliable non-volatile storage (abbreviated RNVS) tier with a RNVS API, and 
 highly reliable non-volatile storage (abbreviated HRNVS) tier with a HRNVS API; 
   presenting to a host system, via an intermediary file system API, a single interface and translating file system requests received via different access protocols into commands compatible with the VS API, the RNVS API, and the HRNVS API, without host system awareness of which of the multiple tiers holds requested data and metadata;   writing, using a write manager, data, received via the intermediary file system API and destined for a file, to the volatile storage tier and marking it to be committed to the reliable non-volatile storage tier;   periodically committing, using a consistency point flush manager, data from the volatile storage tier to the reliable non-volatile storage tier; and   periodically synchronizing, using a synchronization manager, data from the reliable non-volatile storage tier to the highly reliable non-volatile storage tier.   
     
     
         20 . A non-transitory computer readable storage medium impressed with computer program instructions, the instructions, when executed on a processor, implement a file system that transparently deploys file blocks across multiple tiers of storage, with flushing and synchronizing of data from volatile to reliable and highly reliable non-volatile storage, the file system configurable to carry out a method comprising:
 hosting data across multiple tiers of storage via file system application programming interfaces (abbreviated APIs), including:
 volatile storage (abbreviated VS) tier with a VS API, 
 reliable non-volatile storage (abbreviated RNVS) tier with a RNVS API, and 
 highly reliable non-volatile storage (abbreviated HRNVS) tier with a HRNVS API; 
   presenting to a host system, via an intermediary file system API, a single interface and translating file system requests received via different access protocols into commands compatible with the VS API, the RNVS API, and the HRNVS API, without host system awareness of which of the multiple tiers holds requested data and metadata;   writing, using a write manager, data, received via the intermediary file system API and destined for a file, to the volatile storage tier and marking it to be committed to the reliable non-volatile storage tier;   periodically committing, using a consistency point flush manager, data from the volatile storage tier to the reliable non-volatile storage tier; and   periodically synchronizing, using a synchronization manager, data from the reliable non-volatile storage tier to the highly reliable non-volatile storage tier.