not sure if you have given FreeBSD a chance yet and it has an in-progress jail/vm frontend: https://github.com/AlchemillaHQ/Sylve

These days, you're indeed better off using Illumos/SmartOS to run GNU/Linux zones/VMs, rather than native applications, from what I hear.

So many PHP libraries are just wrappers for some other library. I think that's mostly a strength, but in this case it was clearly a weakness.

The SmartOS release changelog for reference: https://us-central.manta.mnx.io/Joyent_Dev/public/SmartOS/sm...

The first link's broken as it includes the semicolon.

It's close, but there are some missing pieces I think. The way it manages storage pools would fit your use case, if you import a zpool, for example, it will scan the datasets and can figure out what zvols should be attached to which VMs..

but there's also VM config info under `/etc/pve` or something similar. I'm pretty sure that's some kind of FUSE filesystem, it's supposed to be synchronized between cluster members.. you might be able to host that externally somehow. But that'll probably take some effort.

You'll also need to figure out how to configure `/etc/network/interfaces` on boot for your network config. But that's doable.

Would be pretty neat.

Kind of. You can run Talos on Proxmox so I don't see why you couldn't run this, but frankly I'd just install Talos or SmartOS on the metal like god intended.

It depends on what "this" you meant, but in general the ways of netbooting an OS are many and varied. You'd have to declare what kind of root device you ultimately want, such as root on iSCSI.

Personally, I feel that "smartOS does not support booting from a local block device like a normal, sane operating system" might be a drawback and is a peculiar thing to brag about.

You can boot ProxMox VMs via PXE:

https://blog.kail.io/pxe-booting-on-proxmox.html

But why bother? A read-only disk image would be simpler.

Running a number of production services on-premise on a big machine using native zones, a few using LX zones (the built-in Linux compatibilty layer), and a single bhyve zone. Actually, years ago this machine was the very first server we set up when our company was just getting started and for the first few years it ran pretty much everything. Zones were ideal for that, also to allow us to pack more services on less hardware, while having decent separation and everything snapshotting/backupping using ZFS. Nowadays we have a bunch more servers, with varying *nix operating systems (SmartOS, Debian, FreeBSD), as well as macOS and even Windows for some specific CI functions. (:

The global zone works great as a hypervisor if you prefer working over SSH in a real shell, and being able to run a lot of services natively just makes things like memory allocation to VM's and having a birds eye view of performance easier. Being able to CoW cp/mv files between zones because it's actually the same filesystem makes certain operations much easier than with actual VM's. Bhyve works well for the things that need an actual Linux kernel or other OS, at the cost of losing some of the zone benefits mentioned earlier.

Highlighting a few things we today run on SmartOS, grouped by their technology stacks: C (haproxy, nginx, PostgreSQL, MariaDB), PHP (various web apps), Java (Keycloak), Elixir/Phoenix (Plausible, fork of Firezone), Rust (rathole, some internal glue services), Go (Grafana, Consul, Prometheus). Most of those are readily available in the package manager, and a few offer native Solaris binaries which run fine on illumos. Others we do local builds in a utility zone before copying the binary package to the where it actually runs.

On LX zones we also run a number of services without problems, usually because they have Debian packaging available but are not in pkgsrc (for example Consul/Nomad, Fabio, some internal things that was already Linux-specific and we haven't bothered to port yet).

And at home a LX zone also runs Jellyfin just fine. (:

Home stuff was the last holdout for me, but even that has been replaced by Proxmox these days. I used SmartOS for a solid 7-8 years, though, and like it for most of that time.

I couldn't point to any one single major reason that prompted the switch - just lots of small annoyances stemming from the world expecting you to be running Linux instead of Solaris, and once you move away from zones, you lose one of the most compelling reasons for being on SmartOS

Certainly Oxide computer company has some use of illumos still, which is strongly related to SmartOS

I use it. It's not the most practical I'll admit but I like the simplicity (I'm not super experienced with linux server administration so both of them feel similarly foreign, but SmartOS is pretty minimal and has been pretty straightforward to manage), and I won't lie it's a fun gimmick to be running a descendant of SunOS for random household services.

It was acquired by Samsung, which is notoriously bad at open source. But the reason why it quietly faded into the background wasn't that. It was that Joyent's ex Sun people had an annoying elitism that made them not care about working with the community.

I have a personal box I keep updated running some utility zones and a couple VMs. I enjoy the tooling very much but it's so niche that I'm wary of using it for Customer engagements.

I never used Solaris in my real life but I can understand the appeal for people who did.

I think the big hit on Solaris and FreeBSD popularity since then was OpenZFS adopting Linux as their main reference operating system.

I use it for a home server. Zones provides a secure way to have (on one physical machine with one physical network interface) some stuff you can only get to on the local network and some things you can get to over the public internet and some things via internet if you have the right ssh key. Each contained natively from each other. Crossbow firewall provides a nice way to contain traffic securely as well. ZFS let me set up two big external usb drives as a raid array, the resulting zvol can (iirc) have multiple filesystems for use by multiple zones although I only use it from one right now for the lan only zone. That zone shares via SMB to my network so I can use it for backups and media streaming.

I’ve been able to do almost everything in native zones. I had a bhyve zone set up to run a photo related GitHub code base that really needed Linux.

SMF is a joy to use for services and package management with pkgsrc is great. The whole thing just feels very thoughtfully put together.

You can probably achieve all this on Linux with docker and the right iptables (or whatever succeeded it) config I imagine? But on smartos I am using facilities that are integrated deeply into the os going back like 20 years now. I also just prefer the old sun stuff.

I prefer SMF to systemd, mdb is pretty nice, and Real Actual RBAC, and in-kernel smb service.

Oxide uses Helios, their own illumos-based distro: https://github.com/oxidecomputer/helios

They’ve written up their reasoning in this RFD: https://rfd.shared.oxide.computer/rfd/0026#_comparison_illum...

bcantrill gave a great talk many years ago about compute-data locality. would be nice to know if those ideas panned out for some customers, but it seems the world has by-and-large continued to schlep data back and forth.

it's too bad too. The concepts behind Manta were such a great idea. I still want tools that combine traditional unix pipes with services that can map-reduce over a big farm of hyperconverged compute/storage. I'm somewhat surprised that the kubernetes/cncf-adjacent world didn't reinvent it.

I did use it on a project, it was meh, alright? In the end the main cost of our processing wasn’t storage latency but code, and this quite arcane scheduler was a barrier too much for most of our team.

I believe it was removed shortly after i left the project..

> This architecture has a variety of advantages including increased security, no need for patching, fast upgrades and recovery.

SmartOS was developed by Joyent for their cloud computing product, it's primary use case isn't desktop computing. I think the advantages mentioned above were probably a bigger factor than the disk space. I would also guess that PXE would be the standard way to boot in a datacenter, not USB.

OpenSolaris was an official OSS version of Solaris which was essentially Solaris, but developed in the open by Sun.

Illumos started as "remove all close source bits and replace with OSS", after Oracle closed down OpenSolaris, Illumos became a full-on fork and Solaris-like rather than another version of Solaris.

From there, multiple distros were born (because Illumos didn't want to be distro), notably OpenIndiana and SmartOS. OpenIndiana being a general purpose distro of Illumos. While SmartOS went with something like "OS for HCI datacenters"

So it's Solaris > OpenSolaris > Illumos.

I believe SmartOS is a distro of Illumos (in the same way that debian is a distro of linux).

Curious why the separate OS should exist. NixOS + Incus sounds like a good solution for declarative provisioning of hypervisor machines.

Not dead, still going strong under new ownership (MNX).

This looks OT?

judging by https://doc.qubes-os.org/en/latest/_images/qubes-trust-level... it looks very linux-centered.