I ended up on shucking 4x the 14 TB WD Elements Desktop. They contain helium drives, the WD140EDGZ in my case, and are about a third cheaper than 4x the 12 TB WD Red Plus drives (which are air-filled). The shucking was easier than I expected too, and the performance seems very comparable. The warranty is a definite downside (European, so no Magnuson-Moss), but I think I can even get them back in their enclosure should they fail during the 2-year warranty period.
I've put some second hand 256 GB M.2 SSDs in there as boot drives. It was a bit of a struggle to get it to work in a way that failure of one of the drives doesn't hold up booting, combined with LUKS, TPM keys and ZFS on root. Learned a lot about systemd-boot which I have never used before, but feels a lot saner to me than grub ever was. So now I have a large script which debootstraps a Debian based NAS into being.
I noticed that there are a lot of ZFS myths and cargo culting. For example TFA mentions ECC RAM, which in some circles is a must-have because ZFS would wreck your pool during a scrub otherwise, which is a myth. It's also very expensive, especially this year. You also don't need much RAM for ZFS, L2ARC doesn't use much RAM at all, to name a few others.
Still doubting about setting `dnodesize=auto` (which is the default), because there are some horror stories about that [1]. And it seems impossible to find a cloud storage provider with reasonable prices that supports `zfs send`. Rsync.net upped their minimum order to 10 TiB recently, which is far too much for my use case.
That under-states the matter. It is a terrible time, price-wise, to build a NAS.
I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.
(If there were a magical choice between having AI and significantly more expensive storage, and having no AI and some program to dump that investment money into getting and somehow leveraging significantly more available storage, that is.)
Seriously, if arts and creativity is what sets humans apart from other animals, then AI has almost completely displaced our capacity to even consider doing these activities ourselves. People reach for AI when they should be composing a birthday greeting themselves.
People cannot be relied upon to provide accurate birthdays but nobody would suffer the social faux pas of an incorrect birthday congrats note. Nor would they send it to a spam catching email, but rather are guaranteed to send it to a regularly checked address. They know the persons birthday after all.
You can then sell high quality birthday information correlated to contact information to ad agencies.
Fuck this current internet. So dystopian.
Like seriously, outside of some close friends and family, are you sitting there deliberating over a message for your coworker Steve in the Slack chat or a cousin you barely talk to outside of a birthday and Christmas card?
> or that people have almost completely lost the ability to consider making art
They haven't lost it, no, but there's a lot of financial incentive to stop paying people to do it.
Almost? ALMOST!?!?!
If you handed me a button that would make it like LLMs had never existed I'd be slamming that button so hard Sam Altman's clothes would spin around.
Return memory and storage prices to normal, undo the sloppification of ~everything, remove all these annoying "features" that are so useful they have to force them upon people, and make scammers actually have to put in a slight bit of effort, all at the "cost" of real human developers, artists, writers, etc. getting paid for their work....
If this is a hard decision for you, you are the problem.
Which is 100% what this has felt like to work with, all spring and summer.
So we're in a slop phase, it'll pass. The first few years of youtube gave no hints that we'd get stuff like Veritasium.
Ignore the slop and use the tools to create something you never thought you could do. That's what it's for!
They (probably) don't want to stop progress (especially unqualified like this) in general. They'd like a world where LLMs didn't come to exist.
And whether LLMs are progress at all remains to be proven.
I built a 24 TB HDD NAS in a 36 bay chassis for around $1000 all-in: mb, ram, rails, chassis, rack, disks, hba, nic
Awful Watts/TB but the plan is to run it and the GPU rig on solar.
The nice thing about the array being big is that I can just RAID 1 it instead of worrying about tinkering with RAID 5, and leave a drive as a hot spare.
PCIe 2.5G NIC for the uplink, and then it can serve over SMB or iSCSI. The main use case for this thing, incidentally, is it just is a caching proxy that holds Docker images, models off of Huggingface, and so on.
I fell in love with the silly idea of squeezing 10G NIC + 10G writes (to a 2x12 RAID 10 HDD array) on the thunderbolt bus of my 12 year old macbook pro.
Then I got distracted with the server mb and truenas and ZFS.
Can't you get that in a single disk that costs less than a kilodollar?
I had a drive go bad in my 4 year old home-built NAS a few weeks ago, and it cost 2x what it originally did to buy the same capacity drive, and that was going to a grey-market importer on eBay, it was more on "reputable" sites.
Like I’m already giving up two full drives for redundancy (which saved my ass - I recently had two drives fail on me in quick succession — both SSDs from what looks like an identical batch) but then the advice is kinda saying I need to keep at least another drive worth of space free for the pool to perform well and not crap itself. That hurts with current prices for sure.
I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
Well, other filesystems can defragment.
ZFS will definitely degrade write performance gradually from 90% utilization, and will hit a stronger cliff at 95%+. Same with SSD, using a consumer SSD above 90%+ utilization would rapidly degrade its lifetime. The effect will be smaller for very large pools and very large files, but the effect is stil there.
I'm repeating this so that people who set up these drives know what is actually going to happen.
Maybe because your statement about SSDs is incorrect. Consumer SSDs have spare blocks too, and wear leveling prevents the scenario you're describing.
Instead the math can be different e.g. it's ok to lose some data but not all. Therefore you might prefer the unraid approach over zfs where losing more than parity doesn't kill the whole pool.
I don't want to lose 26TB just for a mirror, and I have a spare 8TB USB HDD. I'm torn between unraid and just Debian, and I'm torn between just two separate devices and one RAID 1 partition one RAID 0
Technically true.
Practically? There are several other major factors - such as the quality of your drives, whether you periodically run zfs scrubs and/or disk read tests, and how long a resilver takes to complete.
Behind those are the eternal duo - size of budget and cost of failure.
This is a common claim, but honestly, citation needed.
We can't just apply bit error rates from the datasheet that haven't been updated in 15 years. I'm sure a 2-day rebuild is a little more risky than a 4-hour rebuild, but I'm not convinced it's by all that much. Especially if you had a monthly scrub going to prevent disk rot and disks secretly getting super fragile.
> I've had to resilver 2-3 times over the past 10 years, and never lost a byte of data.
And how many times did you have a read error on one of your other disks during a resilver?
> This is a common claim, but honestly, citation needed.
FWIW, here's the quick & dirty argument for that:
Assume:
- We're working within a given generation of disk technology, with comparable drive and interface tech.
- The drives' I/O speed is relatively constant, not scaling with the drive capacity.
- The chance of any given bit (of data on a disk) going bad, in a given time interval, is relatively constant.
- The drives, regardless of size, will be some fixed-ish percentage full. (If nothing else, the boss won't sign off on overspending for unneeded capacity.)
From there - the time needed for resilvering scales linearly with the drive size. As does the number of non-redundant bits that could go bad in any given second of the resilvering. Linear time linear - so the chance of data loss during resilvering rises with the square of the disk size.
We can't just assume this, though. Especially for data that's being checked regularly, because the drive can see how many pre-ECC bits have gone bad and rewrite iffy sectors.
And even if it's true, it's normal to go years without errors. If you have a 15% chance of getting an error each year, and your degraded array has 4 drives without redundancy, then the chance of getting an error in 4 hours is .02% and your chance of getting an error in 48 hours is .3%. Having that risk show up once every few years is pretty negligible compared to everything else that could go wrong with your array.
On top of that, if you don't have a hot spare and it takes 5 days to get a new drive, the difference in resilvering speeds is now 5.2 days versus 7 days and that barely matters.
The main situation where you have to worry about extra drives failing is when you think the resilvering process will greatly accelerate drive deaths. And that depends very much on the specific numbers you have in mind.
This is based on the data I had available 10 years ago when I set up this array… If you have data that points to drives having fewer read errors per GB in recent years, I'd love to see that citation.
> And how many times did you have a read error on one of your other disks during a resilver?
I don't know, because I have RAIDZ2, so that read error would probably be corrected without it being reported to me.
Data you collected or data you found? Any idea where it was?
I've only seen oversimplified math, not real data, and different versions of the math make different assumptions.
> I don't know, because I have RAIDZ2, so that read error would probably be corrected without it being reported to me.
Zpool status gives you a per-disk error count. I kind of assumed someone doing a resilver would use that command.
I think it was a lengthy blog post citing manufacturer data, found via the FreeNAS forums. The added safety of RAIDZ2 was a good tradeoff. Someone else on this thread reports they lost two SSDs in rapid succession, which I hear happens sometimes with drives from the same batch. In any event, I felt no need to launch an independent investigation of the math and underlying assumptions.
> Zpool status gives you a per-disk error count. I kind of assumed someone doing a resilver would use that command.
Yes, I did indeed use zpool to do the resilver… I don't recall if there were any read errors on other drives at the time of those failures, but probably not, as the scrub probably would have flagged it. They've all been replaced with higher capacity drives, so at the moment I see "No known data errors"
Also to reply to your other comment: Personally one extra drive is a small price to pay for the peace of mind.
I achieve peace of mind via backups. For each copy, once I get the AFR below 2% I'm not too fussed about pushing it lower.
[edit] also snapshots aren’t really workable for large files. Remux a movie file and now it occupies twice the space.
Is part of this publicly available? After seeing, - https://words.filippo.io/frood/ - https://0pointer.net/blog/fitting-everything-together.html
I've been looking at setting something up with mkosi/systemd for my home server/NAS - it's interesting seeing how different setups are solving the problem for inspiration.
Does it work? Server can reboot and use TPM to unlock rootfs? What about /boot - encrypted and tamper proof? Resists evil maid attack?
Rabbit hole, I know, but so fascinating if you solved it all :)
But in reality that will never happen and the only actual attack you need to be worried about is junkies breaking in and flogging the drives on facebook marketplace. For which, this level of security is fine.
For this you dont need TPM. Just a LUKS key in rootfs /etc. He said TPM :)
A previous hoster once gave me someone else's drives without wiping them. I don't want random customers snooping around on my data if a similar mistake happens with my disks.
For home use, I run without disk encryption. If I ever need to do data recovery, it's not going to be possible with encrypted disks and one point of a centralized NAS is to have stable long term storage.
> Rabbit hole, I know, but so fascinating if you solved it all :)
Encrypting `/boot/` is the wrong thing to do. One can just replace it with something that logs the encryption key instead (and sends it off to somewhere). You actually need to verify that what you expect to be there is actually there and a encryption isn't gonna do that for you.
A reasonably secure boot chain looks roughly as follows: bootloader is checked by the UEFI (read secure-boot), bootloader chain loads a signed UKI which also uses secure boot verification (or just skip the bootloader if you only have a single kernel with no fallback), this then loads a signed dm-verity `/usr/` which brings you to the login screen. Root is just data (e.g. mounted `noexec`) and is encrypted using e.g. PCR 11 so that it only unlocks if the UKI is the one that is expected (with some other backup unlock method).
This is basically #2 of the "Design Goals" of Poetterings "Fitting everything together"[0]. It's a good read if you are looking to designing your own Linux image and are looking for some inspiration. There is ParticleOS[1] which somewhat dogfoods this kind of system, though I don't think it is anything remotely considered for a production system.
[0]: https://0pointer.net/blog/fitting-everything-together.html
I recommend staying away from SATA drives (huge consumer rush) and look for SAS drives on eBay, particularly HC520 Helium drives from HGST/WD. Need a SAS3008 PCIe adapter ($20) and a SFF-8643 splitter ($30). No backplane is required. Huge lots of decommissioned drives frequently appear. I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
Yea may be some of the stuff is fear mongering and cargo culting. I was told ECC is necessary for ZFS. When the article was written, it was cheap af (2024) to buy ECC RAM so not much consideration was given to it.
-- (*) I have no idea what to do with it. Anyone has any good idea for using 120TB space? I have about 40TB unused bandwidth in the datacenter, may be host a Debian mirror? Donate storage/bandwidth to Internet Archive? Please contact me, appreciate it.
That's cheap indeed. Enough headroom for some failing disks too. How is the noise and power usage? I didn't look at SAS drives at all, because my impression was that they're very noisy. I can place my NAS in a closed off room, but it's not too far away and I was afraid SAS drives would be audible through the wall. At the same time, the shucked drive I'm using presents itself as an WD Ultrastar, which comes very close to a SAS drive, and isn't very noisy.
I think they're same as SATA drives, just different interface. AFAIK they have the same physical dimensions and same internals.
Also something people forget for home nas or figuring out cost for a nas, the power draw of the entire system times your electricity price you then have to multiply this times 2x to 4x times if your climate requires cooling air conditioning of any kind
You have SATA or SAS to pick from. The CPU requirements for a storage server are not high. On a typical ATX board you have motherboard SATA and can put SAS controllers in the spare PCIe slots.
My first "NAS" was two 22TB hard drives in a ZFS pool on my motherboard SATA
https://www.samsung.com/us/memory-storage/nvme-ssd/990-pro-p...
Is it not the case that if you don't have ECC memory, ZFS could end up writing a checksum that does not match the data if you get a bitflip in just the right (wrong) spot?
The myth, popularized by a notorious thread on the TrueNAS forums [1], is specifically that ZFS requires ECC RAM, and will do worse than other filesystems without it, because scrubbing will multiply a single bitflip into a failed pool.
A ZFS core developer says that that isn't the case [2]. Here's some more reasoning [3], also about many other myths.
[1] https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram...
Which means all copies of the data will be corrupted. This can be anything from an irrecoverable file to complete filesystem corruption.
But generally, yeah, not any more dangerous than any other filesystem, and ECC used to be cheap so it was a no-brainer, you should have backups anyway TBH if we're being honest about storage resiliency.
What has been debunked is the "scrub of death" issue, on a scrub a bad bit flip would cause an error, which would be copied over with good data -- well it was technically good before. It would be statistically difficult to have a fault on a read, then a clean read, then a second bit flip destroying the data.
But everything is actually quite sensitive.
We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.
It’s actually less of an issue these days because DDR5 has (by spec) some in-line ECC; won’t help with multi-bit errors but its an improvement on what came before.
AMD has been allowing ECC on lots of regular hardware for a long time.
People don't tend to buy ECC for desktop use because it costs significantly more (used server ram is/was often cheap... but it often doesn't work in desktop boards), and the performance specs are poor.
My home servers are mostly retired desktops, so they get my old desktop ram and I don't want to pay premium prices for jedec speed ecc ram on my desktops, thanks.
Since DDR5 doesn't include reporting on bit errors (afaik), it likely means much fewer single bit errors, but most experienced errors will be multi-bit. Although, I dunno what proportion of bit errors is on the ram chips and what's on the bus... there's no protection from bus errors.
If there were reporting, you could replace chips with high error rates, but without reporting you'll keep running them until they fail enough to notice.
Often unsupported or untested by the motherboard manufacturer, because the precedent set quite a while ago has stood so strongly.
> because it costs significantly more [...] and the performance specs are poor
Because it's niche. If a large share of the desktop market was using ECC memory, the extra cost would have been 10-12%, and a typical kit would push the clocks and timings at least as much as non-ECC memory.
> Since DDR5 doesn't include reporting on bit errors (afaik), it likely means much fewer single bit errors, but most experienced errors will be multi-bit. Although, I dunno what proportion of bit errors is on the ram chips and what's on the bus... there's no protection from bus errors.
Yeah that sucks. I'm really hoping that DDR6 standardizes a light form of end to end ECC. (Bursts are 24 bits wide and 24 bits deep, so on top of the 512 bits of payload there's a bunch of extra bits that can be used for ECC. If memory chips would just store 4 extra bits per 128, that's enough for reasonable ECC on a per-burst basis.)
I wasn't really around for it, but I appreciate the early penny pinchers that developed fake parity ram. The IBM PC required a parity bit per byte?, but you could add an xor chip to calculate it at read and save out on the expensive ram chip.
I wouldn't mind paying a little more for ECC, what with the extra chip and a little more circuitry, but desktop ECC often starts at 50% more, which I'm way too cheap to pay for.
[1]: https://arstechnica.com/civis/threads/ars-walkthrough-using-...
[2]: https://en.wikipedia.org/wiki/ZFS#2004%E2%80%932010:_Develop...
ZFS without ECC is no more risky than any other file system / software RAID without ECC.
As no one owes you an explanation, it would take you five seconds to Google this and discover:
1. It's been disproven, with one of the original ZFS developers chiming in.
2. The original source of the rumor was a forum post that somehow became canon.
...(after much longer) that it's a rabbit hole with nuance. https://news.ycombinator.com/item?id=18481910
Install avahi-daemon. Samba will automatically register with it to advertise SMB/CIFS to macOS and Linux clients over DNS-SD.
Install wsdd2 so that your server will be auto-discovered by Explorer on Windows 10+ clients with SMB 1.0 disabled, too.
Your Linux hostname is probably lower-case, but by default, Samba publishes a capitalized rendering of the hostname to NetBIOS and Avahi. If this bothers you, set “host-name=something” in the [server] section of /etc/avahi/avahi-daemon.conf, and set “mdns name = mdns” in the [global] section of /etc/samba/smb.conf.
If you have macOS clients, you should enable vfs_fruit in your Samba configuration: https://www.samba.org/samba/docs/current/man-html/vfs_fruit..... There are some compatibility reasons to do this, but mostly it means you can set the “fruit:model” so that your server has a fun icon in the Finder sidebar.
To avoid the creation .DS_Store files, you can disallow them: https://ryanoberto.github.io/blog/2015/04/01/disabling-the-c.... I think you can also set “fruit:resource = xattr” to store Finder preferences in xattrs, but I haven't tried it.
Although macOS deprecated AFP in favour of SMB years ago (and are slated to remove AFP client support altogether in the upcoming macOS 27), SMB client support in macOS is still pretty miserable. The upcoming macOS 27 is set to drop AFP support, but until then I will continue to run Netatalk side-by-side with Samba. Netatalk also registers itself with Avahi, and macOS will (tellingly) use AFP preferentially to SMB, so clients will talk to the right daemon automatically.
[public]
vfs objects = catia fruit streams_xattr
fruit:encoding = native
comment = Public share
path = /home/files
force user = files
force group = files
guest ok = yes
browseable = yes
read only = no
create mask = 0644
directory mask = 0755
readdir_attr:aapl_max_access = no
Sure, I traded the convenience of kitchen sink (or Swiss army knife for the more charitable souls) ZFS for some initial pain, but I'm very happy with my choice today.
https://world-playground-deceit.net/blog/2025/06/nas-setup-l...
I've seen the quadlet-nix project but don't get what additional it gets me.
Thanks for saving me probably some hours...
EDIT: I'm also coming from an old Synology NAS from the ~2015 era.
(And I know I have to do that, because when the disk fails it beeps and lights a led near the bad disk)
It’s easy to build a NAS such as the one described in this article, but in the long run, data loss is significantly more likely.
Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.
But what is that command? And how do you know which disk has gone bad?
I am sure I can get an answer from Google / Claude / ChatGPT, but a guide is incomplete without it - and the failure report should be active like a beep or flashing hardware light - I typically log into my NAS only a few times a year. A motd or other banner isn’t sufficient.
I used to look after storage arrays for VFX places.
everytime I saw lvm I inwardly sighed. The docs were terrible, almost as bad as MDADM. snapshots were for a long time unrecoverable. You'd then have to work out what pattern of LV you had, was it a suprise raid0? or a misaligned raid1?
zfs is a night and day compared to LVM/mdadm, two tools, rich help, the man pages are reasonably good, and once you understand zfs vs zpool, you're usually good to go.
https://openzfs.github.io/openzfs-docs/man/master/8/zpool-re...
ZFS got this right. MDADM on ext4 somehow did not.
EDIT: ext4 on MDADM.
How ZFS gets this right: zfs(8) on zpool(8). So, basically, ext4 and mdadm can talk to each other.
schedule a zpool scrub every month, send an email if it finds errors. zpool status will show the errored drive. zpool replace will initiate the drive replacement.
> Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.
`zpool replace my_pool disk3 newdisk`
I use that plus smartctl_exporter so I can do metrics and alerting off that.
"In hashes I trust"
I try to keep my network configuration restrictive by default, so I'm not too concerned about possible security arguments running it from my main machine. I've probably committed some great sin here, but is plain NFSv3 and secondarily Samba (for compatibility) really not enough?
The main reasons people go for ZFS and a "NAS" is checksums and data integrity protection, as well as maybe not wanting to keep their primary machine awake all the time (I personally don't).
Then there are useful features like snapshots, which means I don't have to worry as much about accidentally deleting, or over-writing a file and losing data.
I don't see anything wrong with using a main machine that's up 7x24 as a NAS, don't buy things for the sake of something, but I'm worried about your reliability and bitrot protection. (Yes, it happens, I've seen it first-hand thanks to ZFS).
Filesystems like ext4 are because I value boring and rock solid stability over semi-experimental status of modern filesystems with all the features.
I compulsively look at the rsync backup running to see what's going to transfer, and typically do a dry-run first. That's no perfect solution, but it works for me. If I see something odd I don't quite remember, I check the hashes between both drives, and every time it turns out it was just a path change since I try to keep my data as organized as possible.
rsync arguments are just the plain `--archive --acls --xattrs --verbose` and depending on which backup I'm doing, `--recursive`, `--delete`, `--files-from=`. Nothing other than vanilla at all.
If you were to replace your current drive with a new larger one and your RAM or your SATA controller silently corrupted the files how would things play out? Would these corrupted files propegate to the backups?
Ext4 is battle tested so I understand your reasoning. I think you just need to figure out a way to detect silent corruption and a way to snapshot your files in case they do get corrupted.
From experience I can tell you that it is an absolute pain having to manually sort through a bunch of files trying to detect which ones are corrupt and which ones are good.
It is a matter of time when you’re storing a lot of data — not if.
Even with that, that is still just 2 equally authoritative copies, without any way to know which one suffered the bitrot.
Ok the tie-breaker data could come from history. Yesterdays log and the day before gives you the extra data points to say which side changed. As long as the log itself is somehow above reproach.
Except this script is a myth anyway. You would still need to have something where you tag files as "this file shall never change again, so if it ever appears to, tell me so I can ok or reject the change." and you would have to actually do that tagging and reviewing.
You're right and that's the big thing most are missing. Name your files and write your scripts accordingly I'd say.
My rsync backup'ing script first does a dry run and then serious analyses on the result of the dry run is done. If anything looks out of the ordinary, the script refuses to go ahead and warns me.
rsync silently copying bitrot to backups is the stuff nightmares are made off.
> Even with that, that is still just 2 equally authoritative copies, without any way to know which one suffered the bitrot.
I personally have a sizeable percentage of my files (like, say, family pictures) that have part of the checksum in the name of file:
dsc003784-b3-79a7208c51.jpg
And the system/disk that gave back a bogus checksum has explanations to give ; )
> Except this script is a myth anyway. You would still need to have something where you tag files as "this file shall never change again, so if it ever appears to, tell me so I can ok or reject the change."
I don't think it's a myth. My script does that with the checksum as part of many filenames: if during the dry run the script notices a file with such a checksum is present but different on both systems, I know immediately that something is off.
If you want to keep your setup simple I would consider the following:
- Keep a list of file hashes and check your files against this list periodically.
- Use some type of backup program that supports snapshots. Restic is a good choice.
- Get a hard drive the same size as the one you have and use SnapRAID to manage file integrity or set up a two disk mirror using btrfs.
How do you do the authentication to cockpit, basic HTTP auth in front or something? I've always ended up making my phone SSH-capable rather than dropping down to anything else but public/private keys for authentication to my servers, guess I have a bit of conspiracism when it comes to that.
Building a NAS from scratch is really fun! A small hinderance, but definitely part of the fun as well, is the lack of a "complete resource" on the topic covering how to do every single thing you need to do. Part of the point of my blog post is actually to bring to the internet yet another opinionated NAS setup "guide" (eventhough I would hesitate to call it a guide, but if I ever had to do the same thing again I would definitely read my own post first).
It always gets me how the world of self-hosting is usually introduced from claiming that you can start giving a second life to a Raspberry Pi or a forgotten laptop, and suddenly the next blog you read calls "minimal" a beast machine meant for racks and semi- or professional environments.
Bought a ThinkCentre M910q with an internal SSD and 16 GB of RAM for €200 a couple years ago... Right now I got it chugging along with TrueNAS + 2 USB disks in ZFS Mirror (sitting in a closed cupboard so no chance of cable disconnections).
For me, "minimal home server" means a small computer that fits on a cabinet on the living room, is practically silent, and has a very small power consumption profile (less than a decent Hetzner otherwise the cost wouldn't be worth it). I have a mini-PC in mind, but people think of Dell PowerEdges. Even if given for free, I would never install at my home a PowerEdge for a home server.
I guess it must be the difference between living on a flat vs a 2 story house :-)
I love these machines. I have 3 of them and an M720Q. Needs changed so only one of each is actually in use, but they are so low profile and quiet that I'm considering deploying the other two at home just in case I ever get the urge to play with K8s or something. The only thing I'm missing is a splitter so I can use one plug socket instead of three - not sure if something like that exists.
One nice little fact about the M910Q is that they are rated to maximum 32GB but I have two of them with 2x32GB no problem. Recent RAM prices and forever regret is the only reason the third one doesn't have the same. The M720Q only takes 32GB though.
You can also run 2 SATA off of the M2, and someone created a 3D printable enclosure for them: https://makerworld.com/en/models/1280680-thinknas-2x-hdd-enc...
Is here okay?
I want to have physical storage over here, and logical storage over there, and I want to control the mapping from one to the other. I want to talk about encryption, replication, latency. Make "time machine" available on this logical storage. And then I buy some new physical storage, and it joins the story. This physical storage is over at my friend's house for backup. This physical storage is slow and archival. This is a device for writing archival media, and there's a brand new media in it, go ahead and write to it. Show me the health report on all of the physical media, and show me what you've done to protect the logical storage. Graph my usage and make suggestions about when to add more physical storage.
Buying physical storage with power and wifi, and configuring it with a QR code that's on an e-ink display - seems like it should be the most obvious thing in the world, that we should all be really used to doing by now.
What am I missing?
Next year (post Rust, because networking code is so much nicer in Rust) will be send/recv, and I think we should be able to make some nice improvements over the state of the art there.
> I am creating a RAIDZ1 (RAID 5) zpool. That means 1 drive redundancy in-case of failure
A friend once told me that RAID5 has a high latency cost, because every Write requires a Read to update the stripes across all drives, and while this made sense when drives were expensive, nowadays you might as well do a RAID10 instead, and trade space for latency.
Is this still true with ZFS RAIDZ1?
``` Unlike traditional RAID5 and RAID6 implementations, ZFS supports partial-stripe writes. This has a number of important advantages but also presents some implications for space calculation that we'll need to consider. Supporting partial stripe writes means that in our 7wZ2 vdev example, we can support a write of 12 total sectors even though 12 is not an even multiple of our stripe width (7). 12 is evenly divisible by +1 (3 in this case), so we don't even need any padding. We would have a single full stripe of 7 sectors (2 parity sectors plus 5 data sectors) followed by a partial stripe with 2 parity sectors and 3 data sectors. This will be important because even though we can support partial stripe writes, every stripe (including those partial stripes) need a full set of p parity sectors. ```
I don't have the answer to the latency question, but HDDs have shot up in $/TB over the last couple years too. They are once again kind of expensive.
Please - dont - use - consumer ssd's - with zfs raidz1
I don't know much about ZFS, but it sounds like I need to learn. Docker may have conquered the world, but I plan to stay with LXD for services.
The one thing I take issue with: an appliance like this runs 24/7. It should be low power and fanless. A processor like the N100 seems like the obvious choice.
- Instant, zero-copy container cloning from images via Copy-on-Write. If you boot a new image like the existing ones it's seconds.
- Atomic, millisecond-level instance snapshots regardless of storage size
- Block-level container migration using native 'zfs send' and 'zfs receive', very short command lines and seems to work perfectly.
- Granular dataset nesting (every instance, image, and custom volume gets its own ZFS dataset). You can see every filesystem even on the host.
- Transparent, inline data compression (LZ4/ZSTD) enabled automatically per dataset. For services that don't change much, you might as well use a compressed image to make them even smaller.
- Mirroring / Raid
- Sub-volume sharing and direct management via native ZFS administration tools. If my home directory has a build area and a million files, I can just save time and put my home, pre cooked into a new machine and not copy or even rebuild on my new machine.
- Dedup keeps blocks with the same data as a reference. This costs a lot of memory and has not saved much for me as a lot of my images are similar and already shared I think, but it's cool.
To get around this you can add an SSD mirror as "Special" vdev, the dedup table will then be saved there, freeing up your memory.
But the specs also said ECC RAM and I don't think the N100 supports that.
If I remember correctly it can in theory but in practice I have never seen a N100 with ECC.
Used mergerfs and snapraid, and a simple NFS share. Absolutely perfect for Proxmox backups, our pictures, media, etc, etc. No fuss, easy replacement of drives without needing to keep drives of the same size around.
I'm good with ZFS, have been for years now managing storage for $COMPANY. And I still freaking love using zfs send/receive with proxmox ;-)
But for most at home stuff: mergerfs and snapraid are just more logical.
I am thinking of buying a USB bay with 5 SSD slots in it and then 3 HDD drive. My use case is very, very cold. It is mostly just readonly data with some rclone sync every week.
Does anyone have a suggestion? I am pretty much relying on ZFS to do all the redundancy for me.
We use it for storing (backups, media files), playing (PaperMC), and watching (Jellyfin). I can only complain about the lack of hardware decoding in Raspberry Pi 5. Jellyfin loads CPU much if I enable transcoding so it's always disabled. If I knew this, I'd consider a cheaper and faster, but less popular, RADXA machine. Storage is fast enough for me, rsync and Samba speeds are usually limited by my network. PaperMC also runs without a hitch, thanks for asking!
As I didn't have high requirements for the machine, I also considered USB Bays at first but they wouldn't go well with ZFS.
The biggest concern I'd have with USB is power delivery to the hard drives, but I haven't even done the napkin math so maybe it's fine. The SSDs seem like they might be a waste of money. USB hard drives have a poor reputation, but I don't have a ton of experience to say how much of that is deserved. On a practical level, I'd also be concerned about knocking the cable out.
People say those generate a ton of heat because they're normally in a rack mounted server with plenty of airflow. Probably a significant drawback for a "cheap prebuilt PC".
It's a set and forget OS that will run for years without requiring your attention. But these days it has decent container support for hosting services on.
I had a cloud backup (Backblaze B2, using rclone and encryption) for my home NAS. Some unlucky drive failures later, I was getting ready to recover from my Backblaze backup... and then I couldn't find/remember where I saved the rclone encryption backup.
I lost all data in that NAS, including irreplaceable personal/family photos, due to that mistake. My lesson to share: please verify you can indeed recover from backups.
Also, are there neal.* besides .fun and .computer?
I'll check all the other TLDs real quick...
And that stuff is often not just more expensive, but uglier and noisier. I ended up making my own “enclosure”
I've got a small HP Z440 workstation (small in that it's 10 years old and only single-CPU) and a 14 cores / 28 threads Xeon e5-2680 v4 in it. It's a little workhorse.
I run Proxmox (basically "Debian + ZFS out of the box + easy to spawn VMs / LXCs") on it.
And you can totally put something like FreeNAS in a VM. If you plan to really serve lots of files and need lots of perfs, you should get a tiny bit fancy and passthrough to the FreeNAS VM an entire disk controller (device passthrough to a VM in this day and age of LLMs that are there to find the relevant parts of docs for you in a split second is easy). IIRC the FreeNAS documentation itself explains how to run FreeNAS in a VM so I don't think it's "unsupported".
There's nothing wrong with running it bare either.
They support ECC ram, 4 caddies, one extra PCIe slot, and to my knowledge you're not cpu limited for a zfs file server usecase.
Keep in mind though, all you need is linux* support, iDRAC, ECC if you're a snob, and drive bays ... and that's basically any free server.
In my extremely opinionated opinion I would only get used enterprise server gear, because a zfs file server will just work unless hardware fails. And a UPS.
*ZFS will be a more natural choice on FreeBSD. It's better documented, and will meet Linux 1:1 in hardware compatibility for this.
It works well enough though and has lasted me over a decade at this point. 16GB DDR3 ECC, an old 4 core/8 thread Xeon, 4x14TB drives and the Mellanox NIC.
Throw FreeBSD on it and add a couple lines to /etc/exports and rc.conf and it's a NAS right out of the box
if you are after quietness and "power" then an old workstation is a great bet. They normally have space for at least 4 fullsized HDDs, the more modern ones have lots of PCI space for nvme-ssd cards (for space) and some have lots of lanes for speed
They also tend to come with SAS/whatever remote nvme is called/SATA expnasion options
The down side is they are not as space efficient, they also tend to have 60-120watt CPUs, so expensive to run
Along with various other devices (including a large Synology which I wouldn’t buy today), I run Proxmox on a small two bay+two nvme Terramaster. I have a bare bones Ubuntu LXC running Samba configured for Apple Time Machine, an VM running Scrypyed, and PBS for Proxmox backups. Nothing on it is critical so I don’t bother with any storage redundancy.
Recently replaced the internal USB boot drive with a small NVMe USB enclosure; using a 90-degree USB connector and using a dremel to sand away an opening for the cable to come out so I could mount the enclosure externally.
For some reason people insist on doing truenas on top of proxmox and then introducing a networking layer between everything they do. Noooo…
I’m talking more about people doing the same as you except they’re linking the VM storage to another vm (truenas) over network despite it all being on same host. Think it’s mostly because people don’t want to deal with zfs via command line
The ARM models tend to be very very very slow and small and have architectures that are obsoleted in many Linux distributions combined with hardware that's weird enough that NetBSD doesn't support it (e.g. LAN chip or SATA, which you might need in a NAS). And yes, looking at NetBSD pages and build configs is one avenue for finding well supported devices. Although the documentation isn't great or complete. Some models may also have weird boot sequences or rescue modes that are extra hassle.
I keep wondering if a neat little buildroot config would work with some of these devices. Other odd options would be https://armbian.com/boards and https://openwrt.org/supported_devices .
Caveat: LVM does not have built-in snapshots like ZFS has.
ZFS snapshots use Copy-on-Write differently and have no such allocated space limit. Thus, we can do interesting things like snapshotting a file system after an OS is installed, and then roll back to that snapshot upon an OS upgrade failure, or even clone a new file system from that snapshot to have a different copy of the OS.
Essentially, the nature of the snapshots are different in both.
[1] https://docs.redhat.com/en/documentation/red_hat_enterprise_...
Plus there is sending/receiving snapshots across systems if you want to.
When a ZFS snapshot is marked, then changes to existing blocks in the snapshot as well as new writes all go to free space in the disk. There is no pre-allocated space within which these COW changes are written to.
Thus, we could have a single ZFS snapshot marked today followed by years of data being written to. This is not the intent of an LVM snapshot and the warning that I had quoted above is worth reviewing once.
> 8GB+ of memory for the best performance. It’s perfectly possible to run with 2GB or less (and people do), but you’ll need more if using deduplication.
unraid.net
Android is to Unraid as iPhone is to Synology.
or proxmox ( especially community edition) is fucked or zfs is still unstable, surprisingly with different drives on different interfaces I obtained same results, didnt swap proxmox for clean os, that might being some changes since modules and options would be different i guess. ( drives firmware etc are fine and performing well when inspected with proprietary windows tools)
just search zfs nvme pcie or usb problema or disconnect and you see similar stability problem for different users in different cases (os, drives etc), but also unraid and others, maybe somewhere is rithe right combo of options/glitches but didnt find it yet
That's usually caused by the adapter chipset overheating. Don't use USB adapters for important data.
You're describing ZFS.
Truenas for example add replication, external cloud sync, gui (yes it’s important not to wipe your data with wrong command), HA, custom caching, containers to extend ot with other apps, raid-z expansion with openzfs, and other for monitoring, user management and a lot more.
Other solutions also add more than just a zfs with ssh access.
..running on illumos.