It raised some interesting questions:
- How long can I be productive without the Internet?
- What am I missing?
The answer for me was I should archive more documentation and NixOS is unusable offline if you do not host a cache (so that is pretty bad).
Ultimately I also found out self-hosting most of what I need and being offline really improve my productivity.
I think a cache or other repository backup system is important for any software using package managers.
Relying on hundreds if not thousands of individuals to keep their part of the dependency tree available and working is one of the wildest parts of modern software developmemt to me. For end use software I much prefer a discrete package, all dependencies bundled. That's what sits on the hard-drive in practice either way.
• Info¹ documentation, which I read directly in Emacs. (If you have ever used the terminal-based standalone “info” program, please try to forget all about it. Use Emacs to read Info documentation, and preferably use a graphical Emacs instead of a terminal-based one; Info documentation occasionally has images.)
• Gnome Devhelp².
• Zeal³
• RFC archive⁴ dumps provided by the Debian “doc-rfc“ package⁵.
1. https://www.gnu.org/software/emacs/manual/html_node/info/
2. https://wiki.gnome.org/Apps/Devhelp
There are certain scenarios you have no control over (upstream problems), but others have contingencies. I enjoy working out these contingencies and determining whether the costs are worth the likelihoods - and even if they're not, that doesn't necessarily mean I won't cater for it.
I have long thought that I need my homelab/tools to have hardcases and a low power, modularity to them. Now I am certain of it. Not that I need first world technology hosting in emergency situations, but I am now staying with family for at least a few weeks, maybe months, and it would be amazing to just plonk a few hardcases down and be back in business.
But yeah, things like NixOS and Gentoo get very unhappy when they don't have Internet for more things. And mirroring all the packages ain't usually an option.
I have a bash alias to use wget to recursively save full websites
yt-dlp will download videos you want to watch
Kiwix will give you a full offline copy of Wikipedia
My email is saved locally. I can queue up drafts offline
SingleFile extension will allow you to save single pages really effectively
Zeal is a great open source documentation browser
Unfortunately it doesn't work well on single page apps. Let me know if anyone has a good way of saving those
Selfhosting is a pain in the ass, it needs updating docker, things break sometimes, sometimes it’s only you and not anyone else so you’re left alone searching the solution, and even when it works it’s often a bit clunky.
I have a extremely limited list of self hosted tool that just work and are saving me time (first one on that list would be firefly) but god knows i wasted quite a bit of my time setting up stuffs that eventually broke and that i just abandoned.
Today I’m very happy with paying for stuff if the company is respecting privacy and has descent pricing.
There's your problem. Docker adds indirection on storage, networking, etc., and also makes upgrades difficult as you have to either rebuild the container, or rely on others to do so to get security and other updates.
If you stick to things that can be deployed as an upstream OS vendor package, or as a single binary (go-based projects frequently do this), you'll likely have a better time in the long run.
Single binary sometimes works, but means you need more memory and disk space. (granted much less a concern today than it was back in 1996 when I first started self hosting, but it still can be an issue)
Conflicting versions, I'll give you that, but how frequently does that happen, especially if you mostly source from upstream OS vendor repos?
The most frequent conflict is if everything wants port 80/443, and for most self-hosted services you can have them listen on internal ports and be fronted by a single instance of a webserver (take your pick of apache/nginx/caddy).
> Docker adds indirection on storage, networking, etc.,
What do you mean by "indirection"? It adds OS level isolation. It's not an overhead or a bad thing.
> makes upgrades difficult as you have to either rebuild the container, or rely on others to do so to get security and other updates.
Literally the entire selfhost stack could be updated and redeployed in a matter of:
docker compose pull
docker compose build .
docker compose up -d
You have clean separation between your applications/services and their versions/configurations (docker-compose.yml), and yous state/storage (usually a NAS share or a drive mount somewhere).
Not only are you no longer depended on a particular OS vendor (wanna move your setup to a cheap instance on a random VPS provider but they only have CentOS for some reason?), but also the clean seperation of all the parts allows to very easily scale individual components as needed.
There is 1 place where everything goes. With the OS vendor package everytime you need to check is it in systemd unit? is it a config file in /etc/? wth?
Then next time you're trying to move the host, you forget the random /etc/foo.d/conf change you made. With docker-compose, that change has to be stored somewhere for the docker-compose to mount or rebuild, so moving is trivial.
It's not Nixos, sure. but it's much much better than a list of APT or dnf or yum packages and scripts to copy files around
Isolation technologies are also available outside of docker, through systemd, jails, and other similar tools.
Plus I've found nearly every company will betray your trust in them at some point so why even give them the chance? I self host Home Assistant, but they seem to be the only company that actively enacts legal barriers for themselves so if Paulus gets hit by a bus tomorrow the project can't suddenly start going against the users.
Also an extremely limited list.
People who don’t care, ”I’ll just pay”, are especially affected, and the ones who should care the most. Why? Because today, businesses are more predatory, preying on future technical dependence of their victims. Even if you don’t care about FOSS, it’s incredibly important to be able to migrate providers. If you are locked in they will exploit that. Some do it so systematically they are not interested in any other kind of business.
Also shout-out to Zulip for being open source, self hostable, with a cloud hosted service and transfer between these setups.
Not only that, but it helps to eliminate the very real risk that you get kicked off of a platform that you depend on without recourse. Imagine if you lost your Gmail account. I'd bet that most normies would be in deep shit, since that's basically their identity online, and they need it to reset passwords and maybe even to log into things. I bet there are a non-zero number of HN commenters who would be fucked if they so much as lost their Gmail account. You've got to at least own your own E-mail identity! Rinse and repeat for every other online service you depend on. What if your web host suddenly deleted you? Or AWS? Or Spotify or Netflix? Or some other cloud service? What's your backup? If your answer is "a new cloud host" you're just trading identical problems.
But running it is different issue. Notably, I have no idea, and have not seen a resource talking about troubleshooting and problem solving for a self hosted service. Particularly in regards with interoperability with other providers.
As a contrived example, if Google blackballs your server, who do you talk to about it? How do you know? Do that have email addresses, or procedures for resolution in the error messages you get talking with them?
Or these other global, IP ban sites.
I’d like to see a troubleshooting guide for email. Not so much for the protocols like DKIM, or setting DNS up properly, but in dealing with these other actors that can impact your service even if it’s, technically, according to Hoyle, set up and configured properly.
It's nearly impossible to get 100% email deliverability if you self host and don't use a SMTP relay. It might work if all your contacts are with a major provider like google, but otherwise you'll get 97% deliverability but then that one person using sbcglobal/att won't ever get your email for a 4 week period or that company using barracuda puts your email in a black hole. You put in effort to get your email server whitelisted but many email providers don't respond or only give you a temporary fix.
However, you can still self host most of the email stack, including most importantly storage of your email, by using an SMTP relay, like AWS, postmark, or mailgun. It's quick and easy to switch SMTP relays if the one you're using doesn't work out. In postfix you can choose to use a relay only for certain domains.
The reason why I bring this up is because many early adopters of Gmail switched to it or grew to rely upon it because the alternatives were much worse. The account through your ISP, gone as soon as you switched to another ISP. That switch may have been a necessary switch if you moved to a place the ISP did not service. University email address, gone soon after graduation. Employer's email address, gone as soon as you switched employers (and risky to use for personal use anyhow). Through another dedicated provider, I suspect most of those dedicated providers are now gone.
Yeap, self-hosting can sort of resolve the problem. The key word being sort of. Controlling your identity doesn't mean terribly much if you don't have the knowledge to setup and maintain a secure email server. If you know how to do it, and noone is targetting you in particular, you'll probably be fine. Otherwise, all bets are off. Any you don't have total control anyhow. You still have the domain name to deal with after all. You should be okay if you do your homework and stay on top of renewals, almost certainly better off than you would be with Google, but again it is only as reliable as you are.
There are reasons why people go with Gmail, and a handful of other providers. In the end, virtually all of those people will be better off in both the short to mid-term.
Domains are cheap; never use an email address that's email-provider-specific. That's orthogonal to whether you host your own email or use a professional service to do it for you.
I will lose some email history, but at least I don’t lose my email future.
However, you can’t own a domain, you are just borrowing it. There is still a risk that gets shut down too, but I don’t think it is super common.
I self host my mails but still use a freemail for the contact address for my providers. No chicken and egg problem for me.
I back up all my email every day, independent of my hosting provider. I have an automatic nightly sync to my laptop, which happens right before my nightly laptop backups.
Oh now you don’t only self host, now you have to have space to keep gear, plan backups, install updates, oh would be good to test updates so some bug doesn’t mess your system.
Oh you know installing updates or while backups are running it would be bad if you have power outage- now you need a UPS.
Oh you know what - my UPS turned out to be faulty and it f-up my HDD in my NAS.
No I don’t have time to deal with any of it anymore I have other things to do with my life ;)
Note, I’ve got all the things you mentioned down to the UPSes setup in my garage, as well as multiple levels of backups. It’s not perfect, but works for me without much time input vs utility it provides. Each to their own.
Is it really worth going through so much effort to mitigate that risk?
Cloud is just someone else's computer. These systems aren't special. Yes they are impressively engineered to deal with the scale they deal with, but when systems are smaller, they can get a lot simpler. I think as an industry we have conflated distributed systems with really hard engineering problems, when it really matter at what level of abstraction the distribution happens when it comes to down stream complexity.
How far do we take this philosophy?
Note that I'm not saying you shouldn't self-host email or anything else. But it's probably more risky for 99% of people compared to just making sure they can recover their accounts.
And good luck getting anyone from Google to solve your problem assuming you get to a human.
- kubetail: Kubernetes log viewer for the entire cluster. Deployments, pods, statefulsets. Installed via Helm chart. Really awesome.
- Dozzle: Docker container log viewing for the N150 mini pc which just runs docker not Kubernetes. Portainer manual install.
- UptimeKuma: Monitor and alerting for all servers, http/https endpoints, and even PostgreSQL. Portainer manual install.
- Beszel: Monitoring of server cpu, memory, disk, network and docker containers. Can be installed into Kubernetes via helm chart. Also installed manually via Portainer on the N150 mini pc.
- Semaphore UI: UI for running ansible playbooks. Support for scheduling as well. Portainer manual install.
It's heartening in the new millennium to see some younger people show awareness of the crippling dependency on big tech.
Way back in the stone ages, before instagram and tic toc, when the internet was new, anyone having a presence on the net was rolling their own.
It's actually only gotten easier, but the corporate candy has gotten exponentially more candyfied, and most people think it's the most straightforward solution to getting a little corner on the net.
Like the fluffy fluffy "cloud", it's just another shrink-wrap of vendor lockin. Hook 'em and gouge 'em, as we used to say.
There are many ways to stake your own little piece of virtual ground. Email is another whole category. It's linked to in the article, but still uses an external service to access port 25. I've found it not too expensive to have a "business" ISP account, that allows connections on port 25 (and others).
Email is much more critical than having a place to blag on, and port 25 access is only the beginning of the "journey". The modern email "reputation" system is a big tech blockade between people and the net, but it can, and should, be overcome by all individuals with the interest in doing so.
https://www.purplehat.org/?page_id=1450
p.s. That was another place the article could mention a broader scope, there is always the BSDs, not just linux...
Obviously you should have enough technical knowledge to do a rough sanity check on the reply, as there's still a chance you get stupid shit out of it, but mostly it's really efficient for getting started with some tooling or programming language you're not familiar with. You can perfectly do without, it just takes longer. Plus You're not dependent on it to keep your stuff running once it's set up.