Apparatus and method for managing storage of data blocks

Abstract

A data block storage management capability is presented. A cloud file system management capability manages storage of data blocks of a file system across multiple cloud storage services (e.g., including determining, for each data block to be stored, a storage location and a storage duration for the data block). A cloud file system management capability manages movement of data blocks of a file system between storage volumes of cloud storage services. A cloud file system management capability provides a probabilistic eviction scheme for evicting data blocks from storage volumes of cloud storage services in advance of storage deadlines by which the data blocks are to be removed from the storage volumes. A cloud file system management capability enables dynamic adaptation of the storage volume sizes of the storage volumes of the cloud storage services.

What is claimed is: 
     
         1 . A cloud file storage system, comprising:
 one or more cloud storage services, configured to operate with blade servers, racks and network switches, each of the one or more cloud storage services supporting cloud-based storage of a file system and further configured to store one or more data blocks of the file system at one of a plurality of storage locations as a storage location, the plurality of storage locations comprising at least one storage location with at least one cache memory and at least storage location with at least one non-cache, disk memory, wherein each of the at least one cache memory and non-cache, disk memory has one or more storage volumes;   a cloud file storage management server within the cloud file storage system, the cloud file storage management server further comprising:
 a digital input-output computer interface; and 
 a processor; 
   the processor in the cloud file storage management server configured to execute computer instructions to:
 establish, through a digital communication network, a connection between the cloud file storage management server and a cloud storage service of the one or more cloud storage services; 
 determine the storage location of a data block associated with the file system, wherein the storage location includes at least the cache memory; 
 compute cost information including a cost a cost of storing data; and 
 determine a maximum storage duration specifying a pre-defined maximum length of time for which the data block is to be stored in the cache memory before removal; and 
 analyze, using cost information associated with the one or more cloud storage services available for storing and accessing data blocks of the file system, data-storage and data-access patterns associated with the data block; and 
 based on a data-access pattern of the data block, perform a Least Recently Used (LRU) storage volume replacement algorithm to determine whether and when to remove the data block from the cache memory, prior to an expiration the maximum storage duration, and digitally transfer the data block to the non-cache, disk memory. 
   
     
     
         2 . The cloud file storage system of  claim 1 , wherein the cost information further comprises a bandwidth cost associated with the cloud storage service. 
     
     
         3 . The cloud file storage system of  claim 1 , wherein the cost information associated with the cloud storage service is determined using at least a dynamic pricing model. 
     
     
         4 . The cloud file storage system of  claim 1 , wherein the processor in the cloud file storage management server configured to execute computer instructions to:
 optionally perform an Adaptive Replacement Cache (ARC) storage volume replacement algorithm, in addition to the LRU storage volume replacement algorithm, to determine whether and when to remove the data block from the cache memory, prior to the expiration the maximum storage duration, and digitally transfer it to the non-cache, disk memory.   
     
     
         5 . The cloud file storage system of  claim 1 , wherein the processor is configured to: determine a storage deadline computed for the data block based on the maximum storage duration and a data block eviction time computed for the data block; and
 determine whether and when to remove the data block from the cache memory based on at least one of the storage deadline or the data block eviction time.   
     
     
         6 . The cloud file storage system of  claim 1 , wherein, to analyze storage of the data block at the storage location for determining whether to remove or transfer the data block from the cache memory at the storage location, the processor is further configured to:
 when the data block is stored in the storage location or accessed from the storage location, compute a data block eviction time for the data block using a probability distribution; and   initiate eviction of the data block from the storage location when the data block eviction time elapses without a subsequent access to the data block.   
     
     
         7 . The cloud file storage system of  claim 6 , wherein the processor is configured to:
 determine a data block removal time based on the maximum storage duration;   compute the data block eviction time for the data block using a probability density function configured to describe a probability of removing the data block from the cache memory, the data block eviction time within a range of time from an initial time to the data block removal time and since a last access to the data block; and   initiate eviction of the data block from the cache memory when the data block eviction time elapses without the subsequent access to the data block.   
     
     
         8 . The cloud file storage system of  claim 1 , wherein the data block is stored in a storage volume of the cloud storage service indicated by the storage location, wherein the processor is configured to:
 initiate a resizing event for resizing the storage volume.   
     
     
         9 . The cloud file storage system of  claim 8 , wherein the processor is configured to:
 determine a storage volume size of the storage volume at a time of a previous resizing of the storage volume;   determine a number of replaced data blocks replaced in the storage volume prior to a data block eviction time and since the previous resizing of the storage volume;   determine a number of evicted data blocks probabilistically evicted from the storage volume with the previous resizing of the storage volume;   determine a new storage volume size of the storage volume using the storage volume size, the number of replaced data blocks, and the number of evicted data blocks; and   initiate the resizing event of the storage volume using the new storage volume size.   
     
     
         10 . The cloud file storage system of  claim 9 , wherein the storage volume size at a current resizing event is S, wherein the processor is configured to:
 after the current resizing event, monitor a number of data blocks replaced in S before their respective block eviction times (B replace ) and monitor a number of blocks evicted from S (B evict ); and   at a next resizing event, determine the new storage volume size using S+B replace −B evict .   
     
     
         11 . The cloud file storage system of  claim 10 , wherein the processor is configured to:
 determine a resizing granularity of the cloud storage service indicated by the storage location; and   round the determined new storage volume size to a nearest multiple of the resizing granularity.   
     
     
         12 . The cloud file storage system of  claim 9 , wherein the resizing event comprises:
 an increase resizing events configured to attempt to increase the storage volume size; or   a decrease resizing events configured to attempt to decrease the storage volume size.   
     
     
         13 . The cloud file storage system of  claim 1 , wherein the data block is stored in a storage volume of the cloud storage service indicated by the storage location, wherein the processor is configured to:
 periodically initiate resizing events for resizing the storage volume.   
     
     
         14 . The cloud file storage system of  claim 13 , wherein the resizing events comprise:
 increase resizing events configured to attempt to increase the storage volume size; and   decrease resizing events configured to attempt to decrease the storage volume size.   
     
     
         15 . The cloud file storage system of  claim 14 , wherein a periodic interval associated with the increase resizing events is less than a periodic interval associated with the decrease resizing events. 
     
     
         16 . A method of performing a data block storage management capability, comprising:
 establishing, through a digital communication network, a connection between a cloud file storage management server and a cloud storage service of one or more cloud storage services, each of the one or more cloud storage services supporting cloud-based storage of a file system and further configured to store one or more data blocks of the file system at one of a plurality of storage locations, the plurality of storage locations comprising at least one storage location with at least one cache memory and at least storage location with at least one non-cache, disk memory, wherein each of the at least one cache memory and non-cache, disk memory has one or more storage volumes;   determining a storage location of a data block associated with the file system, wherein the storage location includes at least the cache memory;   computing cost information including a cost a cost of storing data; and   determining a maximum storage duration specifying a pre-defined maximum length of time for which the data block is to be stored in the cache memory before removal; and   analyzing, using cost information associated with the one or more cloud storage services available for storing and accessing data blocks of the file system, data-storage and data-access patterns associated with the data block; and   based on a data-access pattern of the data block, performing a Least Recently Used (LRU) storage volume replacement algorithm to determine whether and when to remove the data block from the cache memory, prior to an expiration the maximum storage duration, and digitally transfer the data block to the non-cache, disk memory.   
     
     
         17 . The method of  claim 16 , further comprising:
 removing the data block from the cache memory, prior to the expiration the maximum storage duration, based on the LRU storage volume replacement algorithm; and   digitally transferring the data block to the non-cache, disk memory.   
     
     
         18 . The method of  claim 16 , further comprising:
 determining a storage deadline computed for the data block based on the maximum storage duration and a data block eviction time computed for the data block; and   determining whether and when to remove the data block from the cache memory based on at least one of the storage deadline or the data block eviction time.   
     
     
         19 . The method of  claim 16 , further comprising:
 when the data block is stored in the storage location or accessed from the storage location, computing a data block eviction time for the data block using a probability distribution; and   initiating eviction of the data block from the storage location when the data block eviction time elapses without a subsequent access to the data block.   
     
     
         20 . The method of  claim 16 , further comprising:
 determining a storage volume size of a storage volume at a time of a previous resizing of the storage volume;   determining a number of replaced data blocks replaced in the storage volume prior to a data block eviction time and since the previous resizing of the storage volume;   determining a number of evicted data blocks probabilistically evicted from the storage volume with the previous resizing of the storage volume;   determining a new storage volume size of the storage volume using the storage volume size, the number of replaced data blocks, and the number of evicted data blocks; and   initiate a resizing event for resizing the storage volume using the new storage volume size.