Methods to identify, handle and recover from suspect ssds in a clustered flash array

Abstract

A technique predicts failure of one or more storage devices of a storage array serviced by a storage system and for establishes one or more threshold conditions for replacing the storage devices. The predictive technique periodically monitors soft and hard failures of the storage devices (e.g., from Self-Monitoring, Analysis and Reporting Technology), as well as various usage counters pertaining to input/output (I/O) workloads and response times of the storage devices. A heuristic procedure may be performed that combines the monitored results to calculate the predicted failure and recommend replacement of the storage devices, using one or more thresholds based on current usage and failure patterns of the storage devices. In addition, one or more policies may be provided for replacing the storage devices in a cost-effective manner that ensures non-disruptive operation and/or replacement of the SSDs, while obviating a potential catastrophic scenario based on the usage and failure patterns of the storage devices.

What is claimed is: 
     
         1 . A system comprising:
 a processor of a storage system coupled to a plurality of storage devices; and   a memory coupled to the processor and configured to store a storage input/output (I/O) stack executable by the processor, the storage I/O stack configured to periodically monitor results of soft and hard failures of the storage devices, the storage I/O stack further configured to perform a heuristic procedure that combines the monitored results to calculate a predicted failure and recommend replacement of the storage devices using threshold conditions based on current usage and failure patterns of the storage devices.   
     
     
         2 . The system of  claim 1  wherein the storage I/O stack includes a health monitor configured to monitor the storage devices in response to a message from a time service of the storage I/O stack. 
     
     
         3 . The system of  claim 1  wherein the storage devices are solid state drives (SSDs) and wherein the results of the soft and hard failures are errors detected at the SSDs that facilitate a predictive analysis of failure of the SSDs. 
     
     
         4 . The system of  claim 3  wherein the storage I/O stack includes a threshold handler configured to read the detected errors from the SSDs and perform the heuristic procedure to determine whether a respective SSD is predicted to fail. 
     
     
         5 . The system of  claim 4  wherein the threshold handler is further configured to maintain a set of counters for each SSD, and wherein the set of counters is configured to record the detected errors based on a type of detected error. 
     
     
         6 . The system of  claim 5  wherein the heuristic procedure is configured to:
 compare a value of each counter to an associated counter threshold value over an associated interval to predict failure of the respective SSD; and 
 in response to the respective counter exceeding the associated counter threshold value over the associated interval, predict failure of the respective SSD and recommend replacement of the respective SSD. 
 
     
     
         7 . The system of  claim 1  wherein the storage I/O stack includes a policy handler configured to implement one or more policies for replacing the storage devices in a manner that ensures non-disruptive operation and replacement of the storage devices. 
     
     
         8 . The system of  claim 7  wherein the one or more policies include a policy that provides a gradual staged replacement of the storage devices over a predetermined time window. 
     
     
         9 . The system of  claim 7  wherein the one or more polices include a policy that enables replacement of one or two storage devices at a time on a storage array configured to sustain up to two storage device failures until data on the failed storage devices is reconstructed on replacement storage devices. 
     
     
         10 . The system of  claim 7  wherein the one or more policies include a policy that specifies replacement of all of the storage devices on a shelf of a storage array at a same time. 
     
     
         11 . A method comprising:
 executing, by a processor coupled to a plurality of storage devices, a storage input/output (I/O) stack including one or more modules configured to predict failure of the storage devices and establish threshold conditions for replacing the storage devices;   periodically monitoring results of soft and hard failures of the storage devices;   performing a heuristic procedure that combines the monitored results to calculate a predicted failure; and   recommending replacement of the storage devices using the threshold conditions based on current usage and failure patterns of the storage devices.   
     
     
         12 . The method of  claim 11  further comprising monitoring the storage devices in response to a message from a time service of the storage I/O stack. 
     
     
         13 . The method of  claim 11  wherein the storage devices are solid state drives (SSDs) and wherein the results of the soft and hard failures are errors detected at the SSDs that facilitate a predictive analysis of failure of the SSDs. 
     
     
         14 . The method of  claim 13  further comprising:
 reading the detected errors from the SSDs; and 
 performing the heuristic procedure to determine whether a respective SSD is predicted to fail. 
 
     
     
         15 . The method of  claim 14  further comprising:
 maintaining a set of counters for each SSD; and 
 recording the detected errors on the set of counters based on a type of detected error. 
 
     
     
         16 . The method of  claim 15  further comprising:
 comparing a value of each counter to an associated counter threshold value over an associated interval to predict failure of the respective SSD according to the heuristic procedure; and 
 in response to the respective counter exceeding the associated counter threshold value over the associated interval, predicting failure of the respective SSD and recommending replacement of the respective SSD. 
 
     
     
         17 . The method of  claim 11  further comprising:
 implementing one or more policies for replacing the storage devices in a manner that ensures non-disruptive operation and replacement of the storage devices. 
 
     
     
         18 . The method of  claim 17  wherein implementing the one or more polices comprises implementing a policy that provides a gradual staged replacement of the storage devices over a predetermined time window. 
     
     
         19 . The method of  claim 17  wherein implementing the one or more polices comprises implementing a policy that enables replacement of one or two storage devices at a time on a storage array configured to sustain up to two storage device failures until data on the failed storage devices is reconstructed on replacement storage devices. 
     
     
         20 . A non-transitory computer readable medium including program instructions for execution on one or more processors, the program instructions configured to:
 implement a storage input/output (I/O) stack configured to predict failure of the storage devices and establish threshold conditions for replacing the storage devices;   periodically monitor results of soft and hard failures of the storage devices;   perform a heuristic procedure that combines the monitored results to calculate a predicted failure; and   recommend replacement of the storage devices using the threshold conditions based on current usage and failure patterns of the storage devices.