Graham Perrin's List: reliability of hard disk, optical disc and flash media

• field-gathered disk replacement data from a number of large production systems, including high-performance computing sites and internet services sites
• annual disk replacement rates typically exceed 1%, with 2-4% common and up to 13% observed on some systems.
• field replacement is a fairly different process than one might predict based on datasheet MTTF
• replacement rates in our data grew constantly with age
• early onset of wear-out degradation
• assumed not to set in until after a nominal lifetime of 5 years
• one instance of a customer rejecting an entire population of disks as a bad batch, in this case because of media error rates

5 more annotations...

• see also: http://www.tomshardware.com/reviews/hdd-reliability-storelab,2681.html - Graham Perrin on 2011-07-09

• 7 Conclusion
• As a first step
• analyzed disk replacement data from a number of  large production systems, spanning more than 100,000 drives from at least four different vendors, including drives with SCSI, FC and SATA interfaces
• For drives less than five years old, field replacement rates were larger  than what the datasheet MTTF suggested by a factor of 2-10

  • Shame on the manufacturers.
  Add Sticky Note
• For five to eight year old drives, field replacement rates were a factor of 30 higher than what the datasheet MTTF suggested

  • Failure rates a factor of 30 (thirty) higher than the manufacturers' datasheets? Shame on them!
  Add Sticky Note
• a continuous increase in replacement rates, starting as early as in the second year of operation.
• anecdotal evidence that bad batches are not unique to SATA disks

5 more annotations...

• Flash memory comes in several different physical packages
• without an external battery and with no moving parts, are extremely robust
• Unless the chip is actually physically, or electrically damaged, CnW has a very high recovery success rate

1 more annotation...

• Q) Why are disks unreadable

   

  A) There are many answers
• physical damage

• Q This disk is not spinning, can you help?

   

  A The answer is probably no.

• Q Why do files get lost?

   

  A There are more answers to this than days in a year, but typical reasons include

   
   
   
   
  • Operator error
   
  • Accidental deletion
   
  • Operating system failure
   
  • Accidental formatting
   
  • Modifying partitions
   
  • Hard disk failure, or partial failure
   
  • Software crashing
   
  • Power glitch
   
  • Physical shock
   
  • Virus (actually this is very rare)

• There are several reasons for data loss, and below are the the most common ones

   
   
  • Accidental deletion and other operator errors
   
  • Hardware failure
   
  • Physical loss of media, can include theft, fire, flood
   
  • Viruses
   
  • Malicious deletion of data
   
  • Operating system corruption
   
  • Damage after a power cut, or computer failure
   
  • Something unexpected happening

• for a non-repairable system, MTTF is equivalent to the mean of   its failure time
• While MTBF is one of the most widely used metrics in reliability engineering, it is also one   that causes a great deal of confusion
• After restoration, the system may not be as good as new
• In most standards, MTBF is used to represent the mean time before   failure

  • …before failure.
  Add Sticky Note
• Even if a standard defines MTBF as the mean time between failures, it inherently   means MTTF because MTTR is almost negligible

  • …between failures
  Add Sticky Note

3 more annotations...

• The average time to failure for a system that is not repairable
• Mean Time To Failure (MTTF) 
• Reliability  The probability that the item will perform a required function without   failure under stated conditions for a stated period of time. Reliability is significant because it   takes into account time.
• reliability is expressed as a   probability, it is always a value between 0 and 1.
• Unreliability is the complement of reliability. If your reliability   is 0.9, the unreliability is equal to 0.1

3 more annotations...

• Compared to other physical formats, disk is ridiculously fragile and easily corrupted/destroyed even when powered off
• begging for disaster, because it's not designed to work that way

• disk replacement rates are much higher than predicted
• SMART (self-monitoring, analysis and reporting technology) code in hard drives and storage management software—long touted by the industry as the best predictor of disk failure—was mostly a security blanket

  • I have seen S.M.A.R.T. hard disk drive modules fail terribly with _no_ hint, from the hardware RAID into which the modules are plugged, that failure is occurring.

  • I'll vouch for this.
    
    A module that reports S.M.A.R.T. failure in one computer may offer no hint of failure in another computer.
  Add Sticky Note
• MTBF is just a statistical measure. We're not supposed to believe it.

1 more annotation...

• annual disk replacements rates were more in the range of 2 to 4 percent
• as high as 13 percent for some sites
• failure prediction models based on SMART parameters alone are likely to be severely limited in their prediction accuracy, given that a large fraction of our failed drives have shown no SMART error signals whatsoever

1 more annotation...

• The current MTBF model isnt accounting accurately for how drives are handled in the field and how they function inside systems
• IT budgets must include a line item for regular replacement of hard disks

• Annual Failure Rates were quite a bit higher than vendor MTBF specs suggest
• the best the Googlers observed was 1.7% in the first year, rising to over 8.6% in the third year
• If you get one of these errors, replace your drive, but not getting one doesn't mean you are safe. SMART is simply not reliable
• low utilization drives having slightly higher failure rates than high ulization ones.
• failure rates do not increase when the average temperature increases
• observed ARRs [Average Replacement Rate] range from 0.5% to as high as 13.5%. . . . up to a factor of 15 higher than datasheet AFRs. Most commonly, the observed ARR values are in the 3%range
• Drives are mechanical devices and wear out like machines

5 more annotations...

• Everything You Know About Disks Is Wrong
• 20 February, 2007
• Maybe consumer stuff gets kicked around more
• significant early onset of wear-out degradation
• drives just wear out steadily
• Instead replacement rates seem to steadily increase over time
• Drives get old, fast
• 99 Responses

6 more annotations...

• http://www.tomshardware.com/reviews/ssd-reliability-failure-rate,2923-9.html - Graham Perrin on 2011-11-16

• drives look more reliable than what you and I see
• results of accelerated life testing
• Vendors typically look at two types of data
• don’t do a good job of reflecting real world usage
• a drive tester consistently “green lights” a unit that invariably fails in the field
• a miracle if you see them in real life.
• your disk drive might crash without warning at any time
• SMART can warn you about some problems, but miss others, so you can’t rely on it. So don’t
• little correlation between disk workload and failure rates
• don’t worry about doing those daily backups to disk and other I/O intensive work

  • Side note: Time Machine backups to a nearby external drive can sometimes be near-endless and unbearably noisy.
    
    Factors include the mass of data at source and capacity of destination. (The larger and fuller a destination, the greater the time taken for Time Machine, Spotlight and other routines.)
  Add Sticky Note
• failure rates do not increase when the average temperature increases
• SOHO users, consider replacing 3 year old disks, or at least get serious about back up.
• 24 Responses

11 more annotations...

• Failure rates are several times higher than reported by drive companies
• YMMV and often will if you deploy these disks in anything but the mildest of eval / demo lab environments
• consumer-class drives’ personality is determined by firmware that assumes the drive is isolated inside a laptop or desktop and cannot rely upon parity information stored on adjacent (peer) drives to recover from a partial or full error condition. Consequently, consumer drives exhibit “Type A” personalities as they heroically go offline for non-deterministic periods of time (a few seconds or many minutes), to “take charge” of the situation and perform various pre-programmed techniques attempting to resolve bad blocks, media / checksum errors, etc…
• many circumstances not restricted to age, environmentals (NVH), power & cooling, or even electro-mechanical behavior of drive peers within the same array, which can render a drive unusable – and eventually failed

2 more annotations...

• the head literally drags along the disk's surface until the air bearing is established