We're in the process of moving to it, and it does seem to have a lot of small bugfixes flying around, but the maintainer is EXTREMELY responsive. I think we'll just end up doing a bit of testing before upgrading to newer versions.

For our use case (3 nodes, 61TB of NVMe) it seems like the best option out of what I looked at (GarageFS, JuiceFS, Ceph). If we had 5+ nodes I'd probably have gone with Ceph though.

We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s)

I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.

I saw an article here not long about where someone explained they were hosting their Kopia or Nextcloud aver Garage, but I can't find it anymore.

This was going to be my next project, as I am currently storing my Kopia/Ente on MinIO in a non-distributed way. MinIO project going to shi*s is a good reason to take on this project faster than later.

I use garage at home, single node setup. It's very easy and fast, I'm happy with it. You're missing out on a UI for it, but MountainDuck / CyberDuck solves that problem for me.

Yeah, that page is horrendous and looks super sketchy. It looks like a very professional fishing attempt to get unsuspecting developers to download malware.

They have a lot of obviously fake quotes from non-existent people at positions that don’t even mention what company it is. The pictures are misgendered and even contain pictures of kids.

Feels like the whole page is AI generated.

Because they aren't thinking about all the chinese wetware they'd be writing down that decision with.

Well, maybe they are using that experience to build something better this time around? One can hope...

Oh no, I used MinIO once or twice for some unlicensed software.

Should I contact a MinIO salesman to purchase an enterprise license ASAP or is it fine if I license my kids and advent of code solutions under the AGPLv3 license ?

  > you may be violating AGPLv3 if you are using MinIO to build commercial products or using it for commercial purposes

Yeah, this is bullshit. I wish the guy used his own advise and spoke to a lawyer :)

Wait, what's the consensus on this? Are they saying that using object storage over a standard network API which they didn't even create, makes your application a derivative work of the object store?

Or just that the users would need to make minio sources, including modifications, freely available?

I guess that's kind of the big question inherent to the AGPL?

I don't think there would be an issue with removing AGPL contributed code. You can't force someone to distribute something they don't want to. IANAL, but I believe that what (all?) copyright in software is most concerned with is the active distribution of code -- not the removal of code.

That said, if there was contributed AGPL code, they couldn't change the license on that part of the code w/o a CLA. AGPL also doesn't necessarily mean you have to make the code publicly available, just available to those that you give the program to (I'm assuming AGPL is like the GPL in this regard).

So, that I'd be curious about it is -- (1) is there any contributed AGPL code in the current version? (2) what license is granted to customers of the enterprise version?

Minio can completely use whatever license they want for their code. But, if there was contributed code w/o a CLA, then I'm not sure how a commercial/enterprise license would play with contriubuted AGPL code. It would be an interesting question to find out.

I don't see a contributor licensing agreement (CLA), so you may be right.

(I personally choose not to contribute to projects with CLAs, I don't want my contributions to become closed-source in the future.)

I'm not a contributor to Minio. This is its own separate thing.

I do have a separate AGPL project (see github) where I have nearly all of the copyright and have looked into how one would be able to enforce this in the US at some point and it did look pretty bleak -- it is a civil suit where you have to show damages etc. but IANAL.

I did not like the FUD they were spreading about AGPL at the time since it is a good license for end-user applications.

It uses LMDB, so if the object mapping fits in memory that should be pretty optimal for reading, while using the build-in Linux page cache and not a separate one (important for testing use cases). For write/deletes it has a bit of write-amplification due to the copy-on-write btree. I've implemented a separate, optional WAL for this and also a mode where writes/delete can be bundeled in a transaction, but in practice I think the performance difference should not matter.

W.r.t. filesystem: Yes, aware of this. Initially used minio and also implemented the use case directly on XFS as well and only had problems at larger scales (that still fit on a machine) with it. Ceph went into a similar direction with BlueStore (reimplementing the filesystem, but with RocksDB).

What would go in to POSIX compatibility for a product like this which would make it complicated? Because the kind of stuff that stands out to me is the use of Linux specific syscalls like epoll/io_uring vs trad POSIX stuff like poll. That doesn't seem too complicated?

I think the criticism (just interpreting the post, don’t know anything about the technical situation) is that the complication is not necessary/worthwhile.

POSIX can be complicated, but it puts you in a nice ecosystem, so for some use-cases complex POSIX support is not over complicated. It is just… appropriately complicated.

Well I didn't mind when they removed it and certainly I didn't consider their paid version which is way too expensive for most use cases.

The UI was useful when first configuring the buckets and permissions; if you've got it working (and don't need to change anything) you're good to go. Also, everything can be configured without the UI (not so easily of course).

yes

They required a "Community Contribution License" in each PR description, which licensed each contribution under Apache 2 as an inbound license.

Meanwhile, MinIO's own contributions and the distribution itself (outbound license) were AGPL licensed.

It's effectively a CLA, just a bit weaker, since they're still bound by the terms of Apache 2 vs. a full license assignment like most CLAs.

Might be because of this other storage product named that https://github.com/NVIDIA/aistore

About a billion dollars difference in valuation up until the bubble pops.

Looks like AI slop

    Replication

    A trusted identity provider is a
    key component to single sign on.

Uh, what?

It’s probably just Minio but it costs more money.

It can store things for AI workloads (and non-AI workloads, but who’s counting…)

It is also useful to remember that MinIO has historically held to an absurd interpretation of the AGPL -- that it spreads (again, according to them) to software that communicates with MinIO via the REST API/CLI.

I assume forks, and software that uses them will be held to the same requirements.

You can, sort of, sometimes. Copyleft is still based on copyright. So in theory you can do a new license as long as all the copyright holders agree to the change. Take open source/free/copyleft out of it:

You create a proprietary piece of software. You license it to Google and negotiate terms. You then negotiate different terms with Microsoft. Nothing so far prevents you from doing this. You can't yank the license from Google unless your contract allows that, but maybe it does. You can in theory then go and release it under a different license to the public. If that license is perpetual and non-revokable then presumably I can use it after you decide to stop offering that license. But if the license is non-transferrable then I can't pass on your software to someone else either by giving them a flash drive with it, or by releasing it under a different license.

Several open source projects have been re-licensed. The main thing that really is the obstacle is that in a popular open source or copyleft project you have many contributors each of which holds the copyright to their patches. So now you have a mess of trying to relicense only some parts of your codebase and replace others for the people resisting the change or those you can't reach. It's a messy process. For example, check out how the Open Street Maps data got relicensed and what that took.

So they’re pulling an OpenAI.

Start open source to use free advertising and community programmer, and then dumps it all for commercial licensing.

I think n8n is next because they finished the release candidate for version 2.0, but there are no changelogs.

Versity is really promising. I got a chance to meet with Ben recently at the Super Computing conference in St. Louis and he was super chill about stuff. Big shout out to him.

He also mentioned that the minio-to-versity migration is a straight forward process. Apparently, you just read the data from mino's shadow filesystem and set it as an extended attribute in your file.

I really like what I've (just now) read about Versity. I like that they are thinking about large scale deployments with tape as the explicit cold-storage option. It really makes sense to me coming from an HPC background.

Thanks for posting this, as it's the first I've come across their work.

I've done some preliminary testing with garage and I was pleasantly surprised. It worked as expected and didn't run into any gotchas.

This is the winner

That may be what S3 is like, but what the S3 API is is this: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3

My browser prints that out to 413 pages with a naive print preview. You can squeeze it to 350 pretty reasonably with a bit of scaling before it starts getting to awfully small type on the page.

Yes, there's a simple API with simple capabilities struggling to get out there, but pointing that out is merely the first step on the thousand-mile journey of determining what, exactly, that is. "Everybody uses 10% of Microsoft Word, the problem is, they all use a different 10%", basically. If you sat down with even 5 relevant stakeholders and tried to define that "simple API" you'd be shocked what you discover and how badly Hyrum's Law will bite you even at that scale.

It gets complex with ACLs for permissions, lifecycle controls, header controls and a bunch of other features that are needed on S3 scale but not at smaller provider scale.

And many S3-compatible alternatives (probably most but the big ones like Ceph) don't implement all of the features.

For example for lifecycles backblaze have completely different JSON syntax

Last I checked the user guide to the API was 3500 pages.

3500 pages to describe upload and download, basically. That is pretty strange in my book.

Everything uses poorly documented, sometimes inconsistent HTTP headers that read like afterthoughts/tech debt. An S3 standard implementation has to have amazon branding all over it (x-amz) which is gross.

HTTP isn't really a great back plane for object storage.

!!!

I’ve seen a lot of bad takes and this is one of them.

Listing keys is weird (is it V1 or V2)?

The authentication relies on an obtuse and idiosyncratic signature algorithm.

And S3 in practice responds with malformed XML, as you point out.

Protocol-wise, I have trouble liking it over WebDAV. And that's depressing.

I suspect there is no decent "minimal" API. Once you get to tens of millions of objects in a given prefix, you need server side filtering logic. And to make it worse, you need multiple ways to do that.

For example, did you know that date filtering in S3 is based on string prefix matching against an ISO8601/RFC3339 style string representation? Want all objects created between 2024-01-01 and 2024-06-30? You'll need to construct six YYYY-MM prefixes (one per month) for datetime and add them as filter array elements.

As a result the service abbreviation is also incorrect these days. Originally the first S stood for "Simple". With all the additions they've had to bolt on, S2 would be far more appropriate a name.

Like everything it starts off simple but slowly with every feature added over 19 years Simple Storage is it not.

S3 has 3 independent permissions mechanisms.

Objects in Garage are broken up into 1MB (default) blocks, and compressed with zstandard. So, it would be difficult to reconstruct the files. I don't know if that was a recent change since you looked at it.

Configuration is still through the CLI, though it's fairly simple. If your usecase is similar to the way that the Deuxfleurs organization uses it -- several heterogeneous, geographically distributed nodes that are more or less set-it-and-forget-it -- then it's probably a good fit.

Ceph is awesome for software defined storage where you have multiple storage nodes and multiple storage devices on each. It's way too heavy and resource intensive for a single machine with loopback devices.

I've been looking at microceph, but the requirement to run 3 OSDs on loopback files plus this comment from the docs gives me pause:

`Be wary that an OSD, whether based on a physical device or a file, is resource intensive.`

Can anyone quantify "resource intensive" here? Is it "takes an entire Raspberry Pi to run the minimum set" or is it "takes 4 cores per OSD"?

Edit: This is the specific doc page https://canonical-microceph.readthedocs-hosted.com/stable/ho...

ceph depends a lot on your use case

minio was also suited for some smaller use cases (e.g. running a partial S3 compatible storage for integration tests). Ceph isn't really good for it.

But if you ran large minio clusters in production ceph might be a very good alternative.

What if a company change license, drop the project or goes bankrupt?

You shouldn't expect guarantee of any kind.

"For enterprise support and actively maintained versions, please see MinIO AIStor."

Commercial only, they will replace the agpl contributions from external people. (Or at least they will say that)

MinIO still makes no sense, as Ceph is fundamentally already RADOS at its core (fully compatible with S3 API.)

For a proper Ceph setup, even the 45drives budget configuration is still not "hobby" grade.

I will have to dive into the MinIO manual at some point, as the value proposition still seems like a mystery. Cheers =3

they do if they know the shoe size of the person who measured it

  Your brain is full of spiders
  You've got garlic in your soul, Mr. Grinch
  I wouldn't touch you with a
  39 and a half foot pole!