Thank you for all these reports over the years.
Thanks for a decade of amazing statistics and lessons. Enjoy resilvering into retirement!
My most recent drive is a WDC WUH722222ALE6L4 22TiB, and looking at the stats (albeit only a few months of data), and overall trend of WDC, in this report gives me peace of mind that it should be fine for the next few years until it's time for the cycle to repeat.
I am becoming more and more convinced that hard drive reliability is linked to the batch more than to the individual drive models themselves. Often you will read online of people experiencing multiple failures from drives purchased from the same batch.
I cannot prove this because I have no idea about Blackblazes procurement patterns but I bought one of the better drives in this list (ST16000NM001G) and it failed within a year.
When it comes to hard drives or storage more generally a better approach is protect yourself against down time with software raid and backups and pray that if a drive does fail it does so within the warranty period.
I'll toss in on that anecdata. This has happened to me a several times. In all these cases we were dealing with drives with more or less sequential serial numbers. In two instances they were just cache drives for our CDN nodes. Not a big deal, but I sure kept the remote hands busy those weeks trying to keep enough nodes online. In a prior job, it was our primary storage array. You'd think that RAID6+hot spare would be pretty robust, but 3 near simultaneous drive failures made a mockery of that. That was a bad day. The hot spare starting doing its thing with the first failure, and if it had finished rebuilding before the subsequent failures, we'd have been ok, but alas.
Like, 25+ years ago I would've bought hard drives for just my personal usage in a software raid making sure I don't get consecutive serial numbers, but ones that are very different. I'd go to my local hardware shop and ask them specifically for that. They'd show me the drives / serial numbers before I ever even bought them for real.
I even used different manufacturers at some point when they didn't have non consecutive serials. I lost some storage because the drives weren't exactly the same size even though the advertized size matched, but better than having the RAID and extra cost be for nothing.
I can't fathom how anyone that is running drives in actual production wouldn't have been doing that.
Of course, learned experience has value in the long term for a reason.
I rescued what could be rescued at a few KB/s read speed and then checked the serial numbers...
I just get 1/3 Toshiba, 1/3 WD, 1/3 Seagate.
Moving to software storage systems (ZFS, StorageSpaces, etc.) has saved my butt so many times.
I mostly just buy multiple brands from multiple vendors. And size the partitions for mdadm a bit smaller.
But even the same model where it's 2 each from bestbuy, Amazon, newegg, microcenter, seems to get me a nice assortment of variety.
By the time Backblaze has a sufficient number of a particular model and sufficient time lapsed to measure failures, the drive is an obsolete model, so the report cannot really inform my decision for buying new drives. These are new drive stats, so not sure it is that useful for buying a used drive either, because of the bathtub shaped failure rate curve.
So the conclusion I take from this report is that when a new drive comes out, you have no way to tell if it's going to be a good model, a good batch, so better stop worrying about it and plan for failure instead, because you could get a bad/damaged batch of even the best models.
Worked in a component test role for many years. It's all of the above. We definitely saw significant differences in AFR across various models, even within the same product line, which were not specific to a batch. Sometimes simply having more or less platters can be enough to skew the failure rate. We didn't do in depth forensics models with higher AFRs as we'd just disqualify them and move on, but I always assumed it probably had something to do with electrical, mechanical (vibration/harmonics) or thermal differences.
Don’t know about disk batches, though. Took used old second hand drives. (Many different batches due to procurement timelines.) Half of them was thrown out because they were clicky. All were tested with S.M.A.R.T. Took about a week. The ones that worked are mostly still around. Only a third of the ones that survived S.M.A.R.T. have failed so far.
Whats funny is that about a year ago I ended up installing FreeBSD onto the same Microserver and ran a 5 x 500GB mirror for my most precious data. The drives were ancient but not a single failure.
As someone who never played with hardware raid ZFS blows my mind. The drive that failed was a non issue because the pool it belongs to was a pool with a single vdev (4 disk mirror). Due to the location of the server I had to shut down the system to pull the drive but yeah I think that was 2 weeks later. If this was the old days I would have had to source another drive and copy the data over.
Every time I think I might need a feature in a file system it seems to have it.
The drives are spaced apart by empty drive slots and have a 12cm case fan cranked to max blowing over it at all times.
It is in a tower though so maybe it was bumped at some time and that caused the issue. Being in the top slot this would have had the greatest effect on the drive. I doubt it though.
Usage is low and the drives are spinning 24/7.
Still I think I am cursed when it comes to Seagate.
The NVMe drives in my servers have these little aluminium cases on them as part of the hotswap assembly. They manage the temperature differential by using a conductive pad for the controller, but not the flash.
I also have multiple drives in operation in the past decade and didn't experience any failures. However unlike you, I didn't use backblaze's drive stats to inform my purchase. I just bought whatever was cheapest, knowing that any TCO reduction from higher reliability (at best, around 10%) would eaten up by the lack of discounts the "best" drive. That's the problem with n=1 anecdotes. You don't know whether nothing bad happened because you followed "the right advice", or you just got lucky.
Careful... that is 22 TB, not 22 TiB. Disk marketing still uses base 10. TiB is base 2.
22 TB = 20 TiB
And in your particular situation, 3 refurbished WUH721414ALE6L4 are the same total price. If you put those in RAIDZ1 then that's 28TB with about as much reliability as you can hope to have in a single device. (With backups still being important but that's a separate topic.)
What do you mean by cycle?
The amount of time they stay on a single drive.
Note, disk failure rates and raid or similar solutions should be used when establishing an availability target, not for protecting against data loss. If data loss is a concern, the approach should be to use back ups.
But yes, I've done the math. I'm just going with the BB numbers here, and after a few years it adds up. The way I understand "peace of mind", you can't have it with a single drive. Nice and simple.
Whether that's right or wrong, when I talked about 5% failure chance per cycle that's what I meant. And 5% is probably an underestimate.
If you're regularly testing your drive's ability to be heavily loaded for a few hours, you don't have much chance of failure during a rebuild.
Edit: By "some flavor" I mean hardware or software.
The amount of personally generated sensitive data that doesn't fit on a laptop's onboard storage (which should all be backed up offsite as well) will usually fit on like a 12TB RAID-1 pair, which is easier to back up than 40TB+ of movies.
It's not the backing up part that's expensive.
I would not be surprised if you decided to spend the dozen hours of your time after all.
Standard S3 is $23/TB/mo. Backblaze B2 is $6/TB/mo. S3 Glacier Instant or Flexible Retrieval is about $4/TB/mo. S3 Glacier Deep Archive is about $1/TB/mo.
I take it you have ~2TB in deep archive? I have 5TB in Backblaze and I've been meaning to prune it way down.
Edit: these are raw storage costs and I neglected transfer. Very curious as my sibling comment mentioned it.
The average person watches more than 3 hours of TV/video per day, and 1 gigabyte per hour is on the low end of 1080p quality. Multiply those together and you'd need 1TB per year. 5TB per year of higher quality 1080p wouldn't be an outlier.
Is that including ads too? And sports/news?
EDIT: Wait, are these "average person" or "average American?"
https://uk.themedialeader.com/tv-viewing-time-in-europe-ahea...
https://www.finder.com/uk/stats-facts/tv-statistics
https://www.eupedia.com/forum/threads/daily-tv-watching-time...
"In a survey conducted in India in January 2022, respondents of age 56 years and above spent the most time watching television, at an average of over three hours per day."
https://www.medianews4u.com/young-india-spends-96-min-per-da...
For china I'm seeing a bit over two and a half hours of TV in 2009, and more recently a bit over one and a half hours TV plus a bit over half an hour of streaming.
Yes it includes ads, sports, and news.
Personally I don't watch a lot of actual TV but I have youtube or twitch on half the time.
If I have 3 disks to devote to backup, I'd rather have 1 local copy and two remote copies, vs 1 local copy with RAID and 1 remote copy without.
The third copy is in the cloud, write/append only. More work and bandwidth cost to restore, but it protects against malware or fire. So it's for a different (unlikely) scenario.
Probably it also matters if you get a bulky 3.5" HDD when all you need is a small flash chip with a few GB of persistent storage — the devil is in the details but I simply didn't realise this could be a part of the decision process
It seems unlikely to me that in a full lifecycle accounting the spinning rust would come out ahead.
It sounded really comprehensive besides having to make assumptions about standard usage patterns, but then the usage is like 10% of the lifetime emissions so it makes a comparatively small difference if I'm a heavy gamer or leave it to sit and collect dust: 90% remains the same
> If this is really a significant concern for you
It literally affects everyone I'm afraid and simply not knowing about it (until now) doesn't stop warming either. Yes, this concerns everyone, although not everyone has the means to do something about it (like to buy the cleaner product)
It turns out that modern hard drives have a specified workload limit [1] - this is an artifact of heads being positioned at a low height (<1nm) over the platter during read and write operations, and a "safe" height (10nm? more?) when not transferring data.
For an 18TB Exos X18 drive with a specified workload of 550TB read+write per year, assuming a lifetime of 5 years[2] and that you never actually read back the data you wrote, this would be at max about 150 drive overwrites, or a total of 2.75PB transferred.
In contrast the 15TB Solidigm D5-P5316, a read-optimized enterprise QLC drive, is rated for 10PB of random 64K writes, and 51PB of sequential writes.
[1] https://products.wdc.com/library/other/2579-772003.pdf
[2] the warrantee is 5 years, so I assume "<550TB/yr" means "bad things might happen after 2.75PB". It's quite possible that "bad things" are a lot less bad than what happens after 51PB of writes to the Solidigm drive, but if you exceed the spec by 18x to give you 51PB written, I would assume it would be quite bad.
With ZFS on Debian the default is to scrub monthly (second Sunday) and resilvering is not more stressful than that. The entire drive contents (not allocated space) has to be read to re-silver.
Also define "high chance." Is 10% high? 60%? I've replaced failed drives or just ones I wanted to swap to a larger size at least a dozen times and never had a concurrent failure.
Home use is also much more likely to suffer from unexpected adverse conditions that impact all the drives in the array simultaneously.
Disaster = Your DC or Cellar is flooded or burned down ;)
I originally used the Backblaze when selecting the drive I'd build my storage pool around. Every time the updated stats pop up in my inbox, I check out the table and double-check that my drives are in fact the 001Gs.. the drives that Backblaze reports has having 0.99% AFR.. I guess the lesson is that YMMV.
refurbs have terrible reliability... They offer 5 year warranties, yet the replacements they send back have terrible quality......
Buy two from different vendors and RAID or do regular off-site backups.
RAID is not a backup! Do both.
Ideally you use "software raid" or file system with the capabilities do scrubbing and repair to detect bitrot. Or have some sort of hardware solution that can do the same and notify the OS of the error correction.
And, as always, Raid-type solutions mostly exist to improve availability.
Backups are something else entirely. Nothing beats having lots of copies in different places.
(HGST drives -- now WDC -- were great, but those are legacy drives. It's been part of WD for some time. The new models are WDC branded.)
https://www.heise.de/en/news/Hard-disk-fraud-Increasing-evid...
After all, it's not just the buyer getting ripped off; it's also Seagate. A customer paid for a brand new Seagate drive and Seagate didn't see a penny of it.
My Louwrentius stats are: zero drive failures over 10+ years.
Meanwhile, the author (Andy Klein) of Backblaze Drive Stats mentions he is retiring, I wish him well and thanks!
PS. The data on my 24-drive NAS would fit on two modern 32TB drives. Crazy.
For my new NVR, the POE power supply is separated out to a powered switch, the newer CPU can do hardware video encoding, and I used SSD for first stage writing and hard disks as secondary backup. The heat has gone way down. So far things have run well. I know constant rewriting on SSD is bad, but the MTBF of SSD indicates it will be a number of years before failing. It’s an acceptable risk.
But equipment designers keep trying to stuff things into spaces that are too small and use inadequate ventilation.
What are some other examples from other companies of this, besides open source code?
Although I will say that the BackBlaze drive stats articles are a much higher effort and standard of quality than you typically see for this tactic.
https://www.techempower.com/benchmarks/#hw=ph&test=fortune&s...
Edit: Adding a shoutout to the iFixIt teardowns that are also quite informative content:
https://www.ifixit.com/Teardown
Edit 2: Also Lumafield CT scans:
Database benchmarks: https://github.com/ClickHouse/ClickBench (containing 30+ databases) and the new JSON analytics benchmark, https://github.com/ClickHouse/JSONBench/
Plus, the hardware benchmark: https://benchmark.clickhouse.com/hardware/ (also used by Phoronix).
Although a minor pet peeve (knowing this is free): I would have loved to see a 'in-use meter' in addition to just 'the drive was kept powered on'. AFR doesn't make sense for a HDD unless we know how long and how frequently the drives were being used (# of reads/writes or bytes/s).
If all of them had a 99% usage through the entire year - then sure (really?).
Incompatibilities between the drive firmware and the device they're in can cause problems.
Subtle harmonic issues with how the drives are mounted, which might be fine for some drives and disastrous for others.
I've always found the best strategy with mechanical hard drives is to have various brands and models in the same device on RAID.
Drives are interchangeable for a reason. :)
1. catastrophic flood in my apartment when drive was on the ground
2. a drive in an external enclosure on the ground that I kicked by mistake while it was spinning
I'm glad I've never had y'all's problems.
That said, my four 2Tb Barracudas still going fine after many years (10+). One failed, replaced with a green. Big mistake, that failed quickly and I went back to standard Barracudas.
They don't get used intensely though.
Yet I've got Toshibas that run hot and are loud as heck that seem to keep going forever.
I never had problems with Seagate Exos or WD Red or even the WD shucked White Reds.
It’s interesting how different the experiences are, some swear by a specific brand.
I think there's a very small chance that Backblaze will be.
Nothing against them, but it's virtually impossible to compete long-term with the economies of scale, bundling and network effects of the major cloud providers.
I don’t think it is going anywhere.
Can anyone recommend a European based alternative with a roughly similar cost?
Thank you very much!
It may actually be a good thing that it's not replicated. That forces me to really make sure I have a separate backup elsewhere.
I used them when I was in europe but migrated away after I came stateside.
Not a problem for cold-storage/batch jobs of course.
YMMV but OneDrive has been improving a lot. Their web photos browsing is comparable to Google Photos these days.
> I'm sure in the EU it is GDPR-compliant.
Sure ain't, especially not in Germany with the BDS.
Check your FARM logs. It sounds like people who were using the drives to mine the Chia cryptocurrency are dumping large capacity drives as Chia's value has fallen.
I'm basically mulling between going as-is to SSDs in a similar 5x4TB configuration, or just going for 20TB hard drives in a RAID 1 configuration and a pair of 4TB SATA SSDs in a RAID 1 for use cases that need better-than-HDD performance.
These figures indicate Seagate is improving in reliability, which might be worth considering this time given WD's actions in the time since my last purchase, but on the other hand I'd basically sworn off Seagate after a wave of drives in the mid-2010s with a near 100% failure rate within 5 years.
They are noticeable much heavier in hand (and supposedly most use dual bearings).
Combined with selecting based on Backblazes statistics I have had no HDD failures in years
https://www.backblaze.com/blog/enterprise-drive-reliability/
https://docs.google.com/spreadsheets/d/1E4MS84SbSwWILVPAgeIi...
nbdkit can emulate disks up to 2^63-1 which is also the same maximum size that the Linux kernel currently supports: https://rwmj.wordpress.com/2018/09/05/nbdkit-for-loopback-pt... https://rwmj.wordpress.com/2018/09/06/nbdkit-for-loopback-pt...
Perhaps we don't need a single flat address space with byte-addressable granularity at those sizes?
I wonder how an 8 bit byte, 48 bit word system would have fared. 2*32 is easy to exhaust in routine tasks; 2*48 not so much.
Extended 96-bit pointers could address the (rather exotic) needs of things such as distributed HPC workloads, flat byte addressable petabyte and larger filesystems, etc. Explicitly segmented memory would also (I assume) be nice for things like peripheral DMA, NUMA nodes, and HPC clusters. Interpreters would certainly welcome space for additional pointer tag bits in a fast, natively supported format.
Given the existence of things like RIP-relative addressing and the insane complexity of current MMUs such a scheme seems on its face quite reasonable to me. I don't understand (presumably my own lack of knowledge) why 64-bit was selected. As you point out addresses themselves were 48-bit in practice until quite recently.
History is filled with paging schemes in computers (e.g. https://en.wikipedia.org/wiki/Physical_Address_Extension). Usually people do this initially as it allows one to access more space without requiring a change of all software, it is an extension to an existing software paradigm, but once the CPU can just address it all as a single linear space, it simplifies architectures and is preferred.
My pondering is less "why such a large address space" and more "why such a large native word size"? Extra bits don't come without costs.
As long as I'm asking ridiculous questions, why not 12-bit bytes? I feel like a 12/48 system would be significantly more practical for the vast majority of everyday tasks. Is it just due to inertia at this point or have I missed some fundamental observation?
https://www.backblaze.com/cloud-storage/resources/storage-po...
There are other less publicly well known things with 72 to 96 drive trays in a single 'server' which are manufactured by taiwanese OEMs for large scale operators. The supermicro is just the best visual example I can think of right now with a well laid out marketing webpage.
edit: some photos
https://www.servethehome.com/supermicro-ssg-6047r-e1r72l-72x...
Also some serious cooling to avoid the drives in the front cooking the drives in the back (assuming front-to-back airflow).
You may not, but plenty of people do.
edit: They look like this: https://knowledgebase.45drives.com/wp-content/uploads/2019/0... (image from ddg)
Careful not to drop it.
This is what backblaze use themselves currently able to hold 60 drives .
Not a server per se, but you just take one 1U server and daisy chain a lot of those JBOD chassis for the needed capacity. You can have 1080 disks in a 42" rack.
Might be the model of the drives. I use 4TB ones.
Considering we used to call them deathstars, I'm surprised they didn't get rid of the line sooner...
If I can be bothered to power on the antiques, which they are built in.
Which is rare, but it happens.
I abused them really hard when they weren't antique.
Still working every time, so far...
Aside from some mishaps (that don't necessarily impact reliability) with vendors failing to disclose the HAMR nature of some consumer HDDs, I don't think there have been any truly disastrous series in the past 10-15 years or so.
You're more likely to get bitten by supply-chain substitutions (and get used drives instead of new ones) these days, even though that won't necessarily lead to data loss.
ST3000DM001.
