Performance monitoring in a distributed storage system

Abstract

Methods and systems for monitoring performance in a distributed storage system described. One example method includes identifying requests sent by clients to the distributed storage system, each request including request parameter values for request parameters; generating probe requests based on the identified requests, the probe requests including probe request parameter values for probe request parameter values, representing a statistical sample of the request parameters included in the identified requests; sending the generated probe requests to the distributed storage system over a network, wherein the distributed storage system is configured to perform preparations for servicing each probe request in response to receiving the probe request; receiving responses to the probe requests from the distributed storage system; and outputting at least one performance metric value measuring a current performance state of the distributed storage system based on the received responses.

What is claimed is: 
     
         1 . A computer-implemented method executed by data processing hardware that causes the data processing hardware to perform operations comprising:
 receiving a selection of a target resource of a distributed storage system, the target resource indicating a respective data group among a plurality of data groups;   generating a health probe request comprising a probe profile, the health probe request configured to identify an availability of the target resource of the distributed storage system based on the probe profile;   communicating the health probe request to the distributed storage system;   receiving a response to the health probe request, the response generated without the distributed storage system performing any read or write operations on the target resource; and   generating a health performance metric based on the response from the health probe request, the health performance metric identifying the availability of the target resource.   
     
     
         2 . The method of  claim 1 , wherein communicating the health probe request to the distributed storage system occurs while another client request requests access to the target resource of the distributed storage system. 
     
     
         3 . The method of  claim 1 , wherein generating the health performance metric based on the response from the health probe request comprises determining that the health probe request fails in identifying the availability of the target resource based on the respective data group. 
     
     
         4 . The method of  claim 1 , wherein generating the health performance metric based on the response from the health probe request comprises representing the health performance metric identifying the availability of the target resource by a ratio of health probe request failures to health probe request successes. 
     
     
         5 . The method of  claim 1 , wherein the operations further comprise comparing the health performance metric to a service level objective (SLO) comprising a target value for the health performance metric for the availability of the target resource. 
     
     
         6 . The method of  claim 5 , wherein the operations further comprise:
 determining that the health performance metric does not meet the target value; and   communicating an indication that the health performance metric does not meet the target value.   
     
     
         7 . The method of  claim 5 , wherein a service level agreement (SLA) for the distributed storage system comprises the service level objective. 
     
     
         8 . The method of  claim 1 , wherein the probe profile comprises a statistical approximation of one or more requests described health probe request. 
     
     
         9 . The method of  claim 1 , wherein the data processing hardware is co-located with the distributed storage system. 
     
     
         10 . The method of  claim 1 , wherein communicating the health probe request to the distributed storage system occurs at regular intervals. 
     
     
         11 . A system comprising:
 data processing hardware; and   memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
 receiving a selection of a target resource of a distributed storage system, the target resource indicating a respective data group among a plurality of data groups; 
 generating a health probe request comprising a probe profile, the health probe request configured to identify an availability of the target resource of the distributed storage system based on the probe profile; 
 communicating the health probe request to the distributed storage system; 
 receiving a response to the health probe request, the response generated without the distributed storage system performing any read or write operations on the target resource; and 
 generating a health performance metric based on the response from the health probe request, the health performance metric identifying the availability of the target resource. 
   
     
     
         12 . The system of  claim 11 , wherein communicating the health probe request to the distributed storage system occurs while another client request requests access to the target resource of the distributed storage system. 
     
     
         13 . The system of  claim 11 , wherein generating the health performance metric based on the response from the health probe request comprises determining that the health probe request fails in identifying the availability of the target resource based on the respective data group. 
     
     
         14 . The system of  claim 11 , wherein generating the health performance metric based on the response from the health probe request comprises representing the health performance metric identifying the availability of the target resource by a ratio of health probe request failures to health probe request successes. 
     
     
         15 . The system of  claim 11 , wherein the operations further comprise comparing the health performance metric to a service level objective (SLO) comprising a target value for the health performance metric for the availability of the target resource. 
     
     
         16 . The system of  claim 15 , wherein the operations further comprise:
 determining that the health performance metric does not meet the target value; and   communicating an indication that the health performance metric does not meet the target value.   
     
     
         17 . The system of  claim 15 , wherein a service level agreement (SLA) for the distributed storage system comprises the service level objective. 
     
     
         18 . The system of  claim 11 , wherein the probe profile comprises a statistical approximation of one or more requests described health probe request. 
     
     
         19 . The system of  claim 11 , wherein the data processing hardware is co-located with the distributed storage system. 
     
     
         20 . The system of  claim 11 , wherein communicating the health probe request to the distributed storage system occurs at regular intervals.