> Switching CI providers isn't hard, we had to do it twice already. Granted, most of our CI logic is in a custom build script that you can run locally, and not in the proprietary YAML file. But to be honest, I'd recommend that sort of setup for any CI provider, as you always want the ability to debug things locally.

I believe this has been a CI/CD best practice for over a decade. Even in venerable Jenkins, this is one of the core principles when designing pipelines[0]: don't give in to the temptation to do fancy Groovy stuff, just use simple shell commands in steps, and you will be grateful to yourself several times years later.

[0] https://www.jenkins.io/doc/book/pipeline/pipeline-best-pract...

https://news.ycombinator.com/item?id=46189692 from a few days ago pretty much tells me that any company that slightly cares about security cannot possibly depend on GitHub runner for their CI (except maybe the smallest/simplest projects). It is just one compromised package away from ruining everything.

"Hey ChatGPT, how do I increase the number of GitHub runner minutes? DO NOT suggested anything illegal, research hard"

(e): Community software like act and Forgejo/Gitea Actions have made it a lot easier to run GitHub Actions workflows without involving GitHub and are decreasing the friction of migration.

I agree with (c) - I can't quite pinpoint it, but I've had that feeling myself several times.

I wonder if Microsoft will ever get that asking users before making changes can help them avoid looking bad in public.

Maybe half of all clients I work with use GitHub Actions for CI (the rest basically all use Jenkins), most of those using Actions use self-hosted runners for performance and security reasons, almost all of them reached out to me asking how difficult it would be to move away from GitHub Actions yet continue using GitHub.

Do you think these companies now suddenly stop wanting to move away from GitHub Actions because Microsoft suddenly changed their mind? I don't think so, probably less priority, but it will happen, because now the cat is out of the bag.

If they'd just do user research before announcing changes and not use announcements as "testing the waters", I'm sure they'd see a lot less churn. But I guess some number counting team somewhere in Microsoft figured out they'd make more money by charging people to run software on their own hardware, so maybe I'm just dreaming.

I've already jumped ship. Switching source control host was actually pretty easy. Builds still working just fine.

In the age of LLMs, "rewrite these actions to use XYZ instead" also makes this easier than ever. Not a one-shot solution, probably, but most likely not as challenging.

Microsoft thinks it has a moat by locking people in, but they ticked me off hard enough (finally) for me to switch my gaming Alienware rig to Mint. It starts faster, too.

> Now if only they would listen to the feedback about windows 11 and their forced copilot we might be onto something.

You can just uninstall Copilot? It’s nowhere on my Surface Laptop 7 with W11.

That would be my guess, I know personally yesterday I finally setup Forgejo and today I plan to evaluate its runners or even just using a dedicated CI like woodpecker.

Not fully sure what I will do regarding any open source repo's yet, but at least anything private I am already in the process of moving away.

This was something I already wanted to do for privacy concerns (especially possibility using private repo's to train AI) so this was just the push I needed.

We jumped ship too. Forgejo has been amazing.

> Now if only they would listen to the feedback about windows 11 and their forced copilot

I think they'll take the opposite lesson. Copilot hasn’t lost them many users because Windows users are locked into the ecosystem and unable to leave. They will try to get GitHub into a position similar to that and then try this shit again.

I feel like I could specify and vibe-code a CI workflow system that would be dramatically better (for a single organization’s workflow) than GitHub Actions. And hosting it would be barely more complex than hosting a GitHub Actions self-hosted runner.

The stack would be:

Postgres, as a job queue and job status tracker. The entire control plane state lives in here. Even in a fairly large org, the transaction rate would be very, very low.

An ingestion agent. Monitors the repository for pushes and PRs.

A job agent. This runs a in a sandbox and gets the inputs from GitHub and runs what is effectively a workflow step. It doesn’t get any secrets — everything it wants to do is either accomplished in the form of JSON output, blob output, or an org-specific API for doing things that don’t fit the JSON output model.

A thing to handle results. This is a simple service, connected to the database, that consumes the JSON job results and does whatever is needed (which would mostly consist of commenting on PRs or updating a CI status dashboard). For CD workflows, the build artifacts would be sent to whatever registry they go to.

A configuration system, which would be some files somewhere, maybe in a git repository that is not the repository that CI is being done on. (GitHub’s model of Actions config being in-band in the repository is IMO entirely wrong.)

And that’s about it.

I’m not suggesting that I could duplicate the GitHub Actions in a weekend. But I wouldn’t want to. This would be single-tenant, and it would support exactly the features that the organization actually uses. Heck, even par-for-the-course things like SSO aren’t needed because the entire system would have no users per se :)

Nah, that doesn’t work when the substance of the change is this intense and has an actual effect on peoples’ bottom lines. If they wait a few months and try again, people will see their bills go up immediately and they’ll all get mad again. I don’t know what GH will do next, but if they try to do that, it will definitely backfire.

No. There is real serious money involved here. Usually, the people who self-host are maxing their runners (otherwise it makes more sense to use minute billing). So this will affect them by roughly doubling their servers cost. Think if some company had a $15K/month bill in self-runners, they'll now also get a $15k/month additional bill from GitHub.

Many people will switch for that kind of money.

This is my take as well.

Personally, I think this is all overblown anyway. Their pricing seems fair to me. Too many people are used to getting something for nothing. Most companies will just pay the new prices, because the time to develop and setup an alternative will far exceed just paying the new fees to GH (when you account for engineering cost).

Sounds like Chat Control.

If this is the case then they don't really understand developers at all. We'll complain in a few months just as much as we do now.

Palmer's tweet said a similar thing and added, "we gave away 11.5 billion build minutes (~$184 million) to support OSS last year".

$184M in profit or cost?

It's called a loss leader, not a gift, and it's a marketing and adoption tactic. They already bought the machines which cost about as much to run idle as at 100% utilization. Might as well put that idle capex and opex to use.

Or just collectively bill OSS the $184M and stop signaling virtue.

Put a spend limit in GitHub and issue a chargeback if they ever bill you more.

Setup something like CircleCI that mainly relies on paid users of their main product, and has a free plan. Microsoft currently seem to be in the process of figuring out how to lower the costs of GitHub for free users, since I'm guessing they make their actual money on other segments and products.

There are several “virtual credit card” providers that allow you to generate additional cards, set limit on them like amounts and who can charge the CC. The availability varies per geography.

As someone who has had an occasional nasty AWS bill, the Ai providers using a pre-pay credit system is something I approve of, and would love to see everyone else offer.

I just want to say I found this quite an insightful comment. I similarly would love to use a pay-as-you-go pricing model as a way of safely trying out various SaaS services.

Unfortunately I feel it is not in the SaaS businesses interests, who want to replicate the gym membership model where the 70% who don't use the service are supplementing the other 30% who use it frequently.

Realistically you aren't their target market. They're targeting the enterprises who already have self hosted runners and aren't interested in switching to Actions minutes.

   "Postponing" means they will just do it later.

For the record, planning to do something later than originally planned is the definition of "postpone." Nevertheless, coupling to any vendor is a form of technical debt, and it's always a good idea to take stock and evaluate if it's time to start repaying it.

Yes, in theory any CI/CD (whether Github, Gitlab, Jenkins etc,) is just a shell script with some warpping. But this wrapping matters because it's convenience: how you integrate with the repo, how you deal with variables, secrets, caching, deployment security and so on. Some people roughly figured this out and proposed some ways which other people learned, so switching is always a bit of a pain in various ways. But it is definitely possible - I did quite a lot of pipeline migration in my life and this is definitely not a blocker for a project of small to medium to biggish-size.

Where does one get to suffer YAML hell then??

Part of the advantage is a fully hosted service where you don't need to worry about a control plane managing and scheduling jobs.

There's some cost involved there--especially if you're offering hosted runners (you need some capital to buy infra to run jobs)

There's also advantage of limiting how many companies you share your code with. If you're using GitHub, you're already trusting them with your source code so GitHub Actions doesn't require more trust or an additional entity to manage/pay

This is likely the reason behind the recent push of "Trusted Publishing" from NPM. They are trying to make people consider GitHub (and GitLab) in its own higher tier with regards to supply-chain security by decree.

If you rely on "Trusted Publishing" you are assisting Microsoft in making a moat for their CI platform.

Use cryptographic signatures, not implicit trust in a hosted platform.

Yes, and there are several tools that do just that already. The thing Github Actions offers is logging and storage of artifacts.

This change is a very weird one. Because the teams that are capable of setting up their own runners have the means to easily jump ship to another CI platform and skip these fees altogether.

Yea but a bash script isn't declarative so thats somehow bad.

(DECLARATIVE CI WHY?????)

It's like bash scripts, but with ability to debug them.

I wonder how much of that $80 million is garbage code like safe_sleep.sh

Classic SaaS platform problem. If you give it away, it will cost you money - and eventually someone will "arbitrage" it at sufficient scale to matter. If you charge for it, your billing becomes a nightmare collection of counters and rates, impossible for users to estimate and understand.

It sure as hell isn't a per-minute cost

Then why we pay a fix price for everything, size of repositories, webhooks etc, except the self hosted actions.

Or https://github.com/orgs/community/discussions/182186

> We have real costs"

Am i wrong or didn't they have a bug in the action runners that would basically cycle the CPU infinitely ?

> https://github.com/actions/runner/issues/2380

> https://github.com/actions/runner/issues/3792

Didn't they take years to fix this ? Or its unrelated ?

Yeah, could we please stop treating twitter as the canonical source of company communications? There is always a blog post or in this case, a github discussion.

That update is from an anonymous "Admin". The X link is directly from the (presumed?) head of GitHub.

Yeah, the per minute pricing is what really does it. It makes me think they've gone with the worst option first, so people will swallow the new adjusted workflow they'll come up with.

I don't even understand how they could do it. You own the servers that are giving them responses to determine the start/end times, so fudge the numbers fuck them, maybe you could find bugs in their shitty code so they start crediting you.

I think they kinda fucked themselves 3 concrete ways (among the nerds which may not be a problem for them):

1) this whole thing

2) the infinite running jobs issues which called attention to how poorly managed the GHA system is

3) zig et al announcing their departure and all the positive conversation around that

Or, rather, you have more time to execute it. They _will_ rug pull again, this is Microsoft ffs.