Consistent data storage in distributed computing systems

Abstract

Methods and apparatus for providing consistent data storage in distributed computing systems. A consistent distributed computing file system (consistent DCFS) may be backed by an object storage service that only guarantees eventual consistency, and may leverage a data storage service (e.g., a database service) to store and maintain a file system/directory structure (a consistent DCFS directory) for the consistent DCFS that may be accessed by compute nodes for file/directory information relevant to the data objects in the consistent DCFS, rather than relying on the information maintained by the object storage service. The compute nodes may reference the consistent DCFS directory to, for example, store and retrieve strongly consistent metadata referencing data objects in the consistent DCFS. The compute nodes may, for example, retrieve metadata from consistent DCFS directory to determine whether the object storage service is presenting all of the data that it is supposed to have.

1 - 24 . (canceled) 
     
     
         25 . A method, comprising:
 performing, by one or more computing systems of a network-accessible service provider:
 receiving configuration input from a client for a distributed computing system; 
 providing the distributed computing system according to the configuration input, including:
 provisioning one or more compute nodes for the distributed computing system; 
 provisioning a distributed computing file system (DCFS) for the distributed computing system via an object storage service that implements a first client-facing interface; 
 provisioning a DCFS directory for the distributed computing system via a database service that implements a second client-facing interface; and 
 providing, to the one or more compute nodes, access to data objects stored the DCFS according to metadata about the data objects stored in the DCFS directory. 
 
   
     
     
         26 . The method of  claim 25 , wherein:
 the data storage service is not guaranteed to return a latest version of the data objects updated via the first client-facing interface; and   the database service is guaranteed to return a latest version of the metadata of the data objects updated via the second client-facing interface.   
     
     
         27 . The method of  claim 25 , further comprising:
 checking, by one of the one or more compute node, a cache for a data object before accessing the DCFS to retrieve the data object.   
     
     
         28 . The method of  claim 27 , wherein the cache is implemented as a local cache of the one compute node. 
     
     
         29 . The method of  claim 27 , further comprising:
 checking, by the one compute node, the cache for metadata of the data object before accessing the DCFS directory for the metadata of the data object.   
     
     
         30 . The method of  claim 25 , further comprising performing, by an agent implemented in the distributed computing system:
 checking for inconsistencies between the DCFS and the DCFS directory; and   responsive to discovery of an inconsistency between the DCFS and DCFS directory, resolving the inconsistency.   
     
     
         31 . The method of  claim 30 , wherein resolving the inconsistency comprises correcting one or more entries in the metadata stored in the DCFS directory. 
     
     
         32 . The method of  claim 30 , wherein the checking is performed periodically as a background process. 
     
     
         33 . The method of  claim 30 , wherein the checking is performed in response to a command of the client. 
     
     
         34 . The method of  claim 30 , further comprising provisioning a resource instance for the distributed computing system to implement the agent. 
     
     
         35 . A system, comprising:
 one or more computing systems that implement a network-accessible service, configured to:
 receive configuration input from a client for a distributed computing system; 
 provide the distributed computing system according to the configuration input, including to:
 provision one or more compute nodes for the distributed computing system; 
 provision a distributed computing file system (DCFS) for the distributed computing system via an object storage service that implements a first client-facing interface; 
 provision a DCFS directory for the distributed computing system via a database service that implements a second client-facing interface; and 
 provide, to the one or more compute nodes, access to data objects stored the DCFS according to metadata about the data objects stored in the DCFS directory. 
 
   
     
     
         36 . The system of  claim 35 , wherein:
 the data storage service is not guaranteed to return a latest version of the data objects updated via the first client-facing interface; and   the database service is guaranteed to return a latest version of the metadata of the data objects updated via the second client-facing interface.   
     
     
         37 . The system of  claim 35 , wherein at least one of the compute nodes is configured to:
 check a local cache for a data object before accessing the DCFS to retrieve the data object.   
     
     
         38 . The method of  claim 37 , wherein the at least one compute node is configured to:
 check the local cache for metadata of the data object before accessing the DCFS directory for the metadata of the data object.   
     
     
         39 . The system of  claim 35 , wherein to provide the distributed computing system, the network accessible service is configured to implement an agent for the distributed computing system, configured to:
 check for inconsistencies between the DCFS and the DCFS directory; and   responsive to discovery of an inconsistency between the DCFS and DCFS directory, resolve the inconsistency.   
     
     
         40 . The system of  claim 39 , wherein to resolve the inconsistency, the agent is configured to correct one or more entries in the metadata stored in the DCFS directory. 
     
     
         41 . The system of  claim 39 , wherein the agent is configured to perform the check periodically a background process. 
     
     
         42 . One or more non-transitory computer-readable media having stored program instructions that when executed on or across one or more computing systems of a network-accessible service cause the network-accessible service to:
 receive configuration input from a client for a distributed computing system;   provide the distributed computing system according to the configuration input, including to:
 provision one or more compute nodes for the distributed computing system; 
 provision a distributed computing file system (DCFS) for the distributed computing system via an object storage service that implements a first client-facing interface; 
 provision a DCFS directory for the distributed computing system via a database service that implements a second client-facing interface; and 
 provide, to the one or more compute nodes, access to data objects stored the DCFS according to metadata about the data objects stored in the DCFS directory. 
   
     
     
         43 . The one or more non-transitory computer-readable media of  claim 42 , wherein to provide the distributed computing system, the program instructions when executed on or across the one or more computing systems cause the network-accessible service to provide an agent for the distributed computing system, configured to:
 check for inconsistencies between the DCFS and the DCFS directory; and   responsive to discovery of an inconsistency between the DCFS and DCFS directory, resolve the inconsistency.   
     
     
         44 . The one or more non-transitory computer-readable media of  claim 43 , wherein to provide the agent, the program instructions when executed on or across the one or more computing systems cause the network-accessible service to configured the agent to perform the check periodically a background process.