▲throwaway15023 hours ago
[-] It is us, developers, who convinced our management to purchase GitHub Enterprise to be our forge. We didn't pay any heed to the values of software freedom. A closed source, proprietary software had good features. We saw that and convinced our management to purchase it. Never mind what cost it would impose in the future when the good software gets bad owners. Never mind that there were alternatives that were inferior but were community-developed, community-maintained and libre.
The writing is in the wall. First it was UX annoyances. Then it was GitHub Actions woes. Now it is paying money for running their software on your own hardware. It's only going to go downhill. Is it a good time now to learn from our mistakes and convince our teams and management to use community-maintained, libre alternatives? They may be inferior. They may lack features. But they're not going to pull user hostile tricks like this on you and me. And hey, if they are lacking features, maybe we should convince our management to let us contribute time to the community to add those features? It's a much better investment than sinking money into a software that will only grow more and more user hostile, isn't it?
0xbadcafebee15 hours ago
[-] > learn from our mistakes and convince our teams and management to use community-maintained, libre alternatives
Every company I've been at that tried to self-host something like GitLab, later moved to GitHub. Nobody in business cares if it's open source/free software. They care about managed hosting, centralized services, invoicing, etc. DIY is great for hobbyists and the cash-strapped.
And I hope ours does too. We're on Gitlab with runners on AWS at the moment, and the overhead is huge. Thousands of working hours spent on setting up and maintaining the infrastructure (even if it's code and probably a fraction of what our own datacenter would involve), millions in costs, hundreds of jobs spinning up to do jobs.
But also, many hours spent building jobs and the like that are off-the-shelf on Github Actions.
This is the main issue though, it feels like doing anything but what the largest companies offer just costs more time and money. It reminds me of the decades long push to move away from Microsoft, only for e.g. the office 365 offering to show up and make everyone's (software, account management) work easier and cheaper, forever.
Well, what else can we say? Enjoy your unilateral price increases or features shoved down the throat (CoPilot anyone?), with exactly 0 extra money in your pocket. Unless you are a C-level, of course.
reply▲Yeah I can't see a better alternative to GitHub.
OSS can build truly incredible libraries and frameworks. User facing products? ehhhhh not so much.
GitHub has gotten worse over the years but it's not like there's some gold standard open source alternative. And remember, early GH was filled with a pretty amazing group of developers and open source advocates.
If the counter is that, instead of buying github, we could have invested in building some other tool, well, that's not how this works. People need to build what the company is actually building for its users.
I'm a contractor so have worked on a lot of different projects for different companies, big and small and also early startups.
This is far from truth imo. It is very possible to only use (F)OSS. Github, AWS, Azure, Vercel are not at all more pleasant or easier to work with than on-prem Gitlab/gitea/codeberg/jenkins/k8s/kibana/prometheus/grafana.
I could spend an hour and have a full setup done on physical or VPS to have 1) remote git hosting 2) pipelines running on changes 3) pipelines publishing images or some artifacts 4) automated deployment for these images/artifacts
I'm struggling to see what am I missing. What is worthwhile that Github offers? It is popular and easy to set up org+repos, but that seems it.
A few years ago I was working on Azure Devops with the azure pipeline and that was the worst developer experience I've ever had. AFAIK Github actions uses the same syntax and works the same way, at least it was at that time.
anchochilis8 hours ago
[-] What you're missing is maintenance, security, scaling, and protection from data loss.
Bespoke CI is easy to build but no one wants to be in charge of rolling out a critical security patch to that on-prem box no one's touched since that consultant from 2 years ago.
Your CI has to be
fully codified, stateless and possible to redeploy with a single command. That's the only way it can remain sustainable. No persistent hidden state, no manual configs (even as an option!) and automatically rebuilt on every release as the new version is deployed.
As a really big bonus, that also makes your CI testable.
In the previous job, we built such a thing: https://smarketshq.com/building-a-reproducible-ci-system-for...
Yes, totally agreed in theory, and it sounds like y'all built a great solution for your use case. But it takes substantial effort and discipline to do something like that at scale.
At some point, you develop complex interdependencies with other systems. You need sophisticated caching for optimum build performance. Techniques like GitOps are unsustainable at a certain number of engineers/commits per hour.
>I could spend an hour and have a full setup done on physical or VPS to have 1) remote git hosting 2) pipelines running on changes 3) pipelines publishing images or some artifacts 4) automated deployment for these images/artifacts
This sounds like a week or two of work to me (I'm a novice though). You should write a guide.
At work we've started switching to codeberg (which uses forgejo) and honestly it's a breath of fresh air compared to GitHub. It's blazing fast compared to GitHub and has feature parity with our needs.
reply▲TheNewsIsHere1 hour ago
[-] I have (almost completely) moved my business from GitHub to Forgejo as well. I’m deploying an instance at home as well.
It’s shocking how bloated and slow GitHub is even for basic actions when you compare it to Forgejo running just your own stuff.
Bonus: if you can manage to reliably backup a database and a filesystem, you can then backup your forge. Outside GH Enterprise there is still no restorable backup option inside GitHub.
Not an option for the majority of companies as it only allows open source repositories as far as I'm aware.
reply▲homebrewer10 hours ago
[-] Selfhosting gitea is trivial, I'm saying this as someone who has been doing it at work for almost 6 years. Our experience has recently prompted another org (run by people we know) to move off GitHub, they also seem to be happy.
reply▲There are plenty of alternatives to Github, from Gitlab and Gittea to Forgejo, but Codeberg is not one of them, which is what I wanted to stress.
reply▲on the other hand, a better user experience is something open source projects can aspire towards as a pure goal, with no perverse incentives pulling them the other way. a lot of the recent github changes have degraded the user experience because that was overall more profitable for the company.
reply▲officialchicken11 hours ago
[-] Funny how Microsoft products tend to instill blindness over time; refocusing on git hooks (and not the UI) can offer some perspective.
reply▲Forge, Gitea, and git itself contains a cgi script with quite a bit of functionality. And of course, the way it is supposed to work, git-over-ssh, as in give committers Linux shell accounts on a shared machine, with the CGI script running for pretty pictures (Remember CGI? You know, "cloud functions" before such a thing existed)
Huh, I should make an Apache plugin that launches docker exported containers uploaded into a directory.
Why a shared machine? Git was “supposed” to work with email. Do that.
reply▲JoshTriplett3 hours ago
[-] That would be a huge downgrade for most users.
reply▲So is managing a shared Linux central repository…
reply▲Lots of businesses can't put sensitive work product onto external servers. You just don't work for them.
reply▲makeitdouble13 hours ago
[-] > DIY is great for hobbyists and the cash-strapped.
And the companies with specific needs (funnily enough, at the other end of the cash spectrum) or have a lower internal cost than the products out there.
There's more of them out there than we give credit for I think. In particular, running one's own stack on the corner seldom makes the news.
This is the irony of software engineering. This is how a lot of bad software gets written too.
reply▲> Every company I've been at that tried to self-host something like GitLab, later moved to GitHub
Interesting, my experience has been the opposite. 99% of companies I've seen self host their VCS, it has been Gitlab (with some rare sel-hosted GitHub Enterprise everyone seems to hate, and the very rare Bitbucket).
To be fair most of them started with it when Gitlab was really really ahead, features wise. The gap has somewhat closed, but Gitlab is still a superior product IMO. Just the fact that you can have an actual organisational structure, and move it around, and share variables/configs between groupings, beats anything GitHub have to offer which is slightly nicer than GitLab.
Are you based in Europe? Gitlab has a much stronger presence there. In the US Gitlab adoption is much weaker in my experience.
reply▲Yep, but I interact with companies all over the world, and while American ones tend to not self-host (in general, but VCS in particular), out of those that do, Gitlab still seems more popular.
reply▲nacozarina14 hours ago
[-] I’ve setup self-hosted gitlab configs for multiple projects and it works fine.
reply▲How does Forgejo compare here? Would it be better or worse than Gitlab? How about a self-hosted Sourcehut instance?
reply▲> later moved to GitHub
And had to live with their constant outages :)
whateverboat12 hours ago
[-] LOL. You can do GitLab hosting with High Availability at a very professional level without needing a huge team.
reply▲GitHub isn’t even good, it’s just the mediocre default everybody uses. PRs were fantastic and the best thing ever - 15 years ago!
reply▲NamlchakKhandro13 hours ago
[-] 100% don't understand why people think github actions are terrible.
everything else is trash.
Github Actions changed the landscape.
They're composable.
The only two other things that come close is Concourse.CI and CircleCi.... and circle-ci is 100% trash
Github Actions is a cobbled together mess. It is mainly based on Azure DevOps Pipelines and still has some glaring bugs and wildly inefficient parts.
If it works for you, great. But it is far from being good.
We use Gitlab for CI/CD and tbh it is amazing. Simple, predictable, debuggable.
reply▲This whole thread is various people saying "[This] is trash, [that] is awesome", with the next person claiming the opposite. I suspect most people with strong negative opinions here know enough to have felt the pain, and not enough to be able to properly reason about the system.
I've worked with Github Actions, Gitlab-CI and CircleCI in the last 10 years, and they've all been such an improvement over Jenkins, or god forbid, CVS with manual deployments, that I'm generally just counting my blessings.
For me the pain only came when not adhering to KISS. All the mentioned VCS are pretty much feature complete and only really differ on meta-topics (cost, license, lock-in) or niche topics (Actions marketplace, matrix builds, SSH on Runners). I've not yet run into an issue that would have actually blocked me, because there's always sh to fall back to in case of a bug or missing feature.
brian_cunnie1 hour ago
[-] Thumbs up on Concourse CI: I like seeing all my builds at once on any easy-to-read dashboard. That’s why we switched from GitHub actions: the dashboard.
reply▲Versioning sucks (the references are mutable), debugging sucks, you cant run them locally.
reply▲sunnyday_00210 hours ago
[-] Pin the action's version via a digest and use Renovate for updates.
You can run all your CI locally if you don't embed your logic into the workflows, just use CI for orchestation. Use an env manager(Mise, Nix etc) to install tooling(you'll get consistency across your team & with CI) and call out to a task runner(scripts, Make, Task etc).
> You can run all your CI locally
if you can, you don't need CI. we can't (too slow, needs an audit trail)
I think the idea is GitHub actions calls "build.sh", or "deploy.sh" etc. Those scripts contain all of the logic necessary to build or deploy or whatever. You can run those scripts locally for testing / development, or from CI for prod / auditing.
reply▲I think I agree with you that:
- everything else is trash.
- Github Actions changed the landscape.
- They're composable.
And I still hate github actions! Aside from anything else, they have one major flaw, which is there is no good development/test loop for writing them.
If you write most of your CICD in some kind of script, then you can run it locally, and do some basic checks around environment etc before deploying.
If you write most of your CICD in github actions or any alternative, you will be doomed to push 100 commits with messages like "maybe be?", "hmmm. . ." before eventually squashing them all down when it turns out several hours later that you mispelt an environment variable.
top tip: make a repo in your org for pushing all these nonsense changes to, test out your workflows with a dummy package being published to the repo, work out all the weird edge cases/underdocumented features of Actions
once you're done, make the actual changes in your real repo. I call the test repo 'pincushion'
We call ours "bombing-range"
We maintain an internal service that hosts two endpoints; /random-cat-picture (random >512KB image + UUID + text timestamp to evade caching) and /api/v1/generic.json which allows developers and platform folks to test out new ideas from commit to deploy behind a load balancer in an end-to-end fashion, it has saved countless headaches over the years.
GitHub Actions are amazing ! For a public repository who will give you a free machine to run for 6 hours at a stretch for a job !!
reply▲This thread is largely commentary on the
technical aspects of GitHub Actions.
The fact the business gives away free compute is irrelevant and more a discussion of their marketing budget.
Without trying to sound snarky: What is the fancy alternative you are suggesting to pull requests?
reply▲Something like Gerrit. Instead of carefully crafting a logical series of patches that are all well documented with commit messages, PRs are just garbage filled diff soup of "fix typo" commits. I hate it. It's hard to review and seems to be based on putting the least amount of effort into proposing changes to the code. See
https://gist.github.com/thoughtpolice/9c45287550a56b2047c631...That's down to culture and (self) discipline, not tools.
reply▲It's not entirely, because Github simply does not support inter-version diffs when you have multiple commits. If you force push onto multiple commits there is no way to show a diff between version 2 and version 3 of those commits. How Github lacks such basic (and imo necessary) functionality in 2025 is amazing to me.
Something like linked and dependent PRs in a chain would go someway to replicating Gerrit but again this basic functionality is not available out of the box for whatever reason.
sublimefire11 hours ago
[-] Well but this is controllable, i.e. it is people who choose to do this not the platform. Very much an internal design choice.
reply▲Just curious - what do you think the current best git platforms are?
reply▲What ever happened to Hudson/Jenkins? That was the full featured CI/CD solution before github actions.
reply▲shawabawa314 hours ago
[-] I'm not at all a fan of GitHub actions, but come on, Hudson/Jenkins was a nightmare world, GitHub actions is a million times better
reply▲Working with Jenkins CasC, JobDSL and declarative pipelines, I'm not sure where the million times comes from. Sure, there are some annoying parts, and GHA has the social network for reusable actions, but apart from that it's not that different.
Oldschool maven type jobs where you type shell script into a `<textarea>`? Yeah, let's not talk about those, but we don't have a single one left anymore.
Jenkins Groovy is awful and full of footguns. Have you ever run into a serialization exception?
It's too powerful and there are too many of its implementation details exposed to the user.
I haven't seen a serialization exception, but I have run into plenty of footguns with YAML (ref GitHub Actions).
The DSL semantics can be weird with when things like params/env expansions in options block are evaluated.
paulddraper14 hours ago
[-] Jenkins is in every way a “Java” program, and not the good kind.
What you can say for it, is that it was free and near infinitely hackable.
robot-wrangler14 hours ago
[-] That also is/was awful. But it's just another platform like GHA, and the solution to this kind of thing is always the same, should not be surprising, and is boring in the good way. Write automation so that it's not tightly coupled to the platform on the backend. If you can't migrate between platforms then you're eventually going to be unhappy.
If someone is forcing you towards high stakes tight-coupling with no thought whatsoever towards the lock-in, you should get it in writing that "we at ${org} are fully committed to ${vendor} with ${platform}, on ${cloud} using ${tech} come what may, now and forever" and lots of sign off so that everyone knows who to blame when this is inevitably wrong.
aetherspawn16 hours ago
[-] Azure DevOps is the gold standard for Pipelines.
reply▲That's not saying much, since it's still dependent upon the untyped mess that is YAML.
reply▲0xbadcafebee15 hours ago
[-] YAML is just a data format. Make your own "thing" that takes input in any format you desire, then dump it to YAML. (also, YAML is dynamically typed, and supports explicit typing, but the parser can choose to ignore it)
reply▲Really?
I grant you pipelines are the best bit about ado, but the fact that you can’t test them is a pain.
And the webhooks and templating are pretty messy and unpleasant quickly.
We’re changing from ADO to GitHub (had to be an MS product for corporate) and the infra people are looking forward to GHA as they prefer their maintenance to ADO pipelines.
if only they supported ed25519 ssh keys
reply▲AD is just sourcesafe/tfs/vsts with rebranding, each time trying to get rid of the bad reputation in developer circles.
reply▲Not to defend this behaviour, but a lot of clouds SaaS do require you to pay for both ”management” and for the actual resources. And if you’re using vms in their cloud, you pay twice.
For example, Azure has had a script runner service for ages that you can hook up to your ”own” vm, by installing an agent. But then you pay double, the fee (per second) for the service as long as the script is running, and the fee (per second) for the vm in azure as long as it exists. So, as with GitHub actions, it’s cheaper to run it on their provided crap instances.
To get rid of the double costs I guess you could install your own CI server and agents, that polls the GitHub repo, but then you don’t get the integration in their web gui. That was what you did before gh actions came around, a local Jenkins for example.
> alternatives that were inferior
Actually there were alternatives that were far superior (seriously, no way to group projects?) but also more than 2x as expensive. If GH "fixes the glitch" then it will be plan B time.
What are these superior alternatives? Never could stand GitHub and I'm suffering GitLab at the moment.
reply▲physicsguy10 hours ago
[-] There's always been this lesson with CI/CD - don't couple yourself to a specific product. If you do, you're gonna get screwed eventually. It happened with TravisCI, CircleCI, now it's happening with GitHub. The business model only makes sense if you can charge for it, those charges are only ever going to go up.
reply▲It's not my money man. It's still fine.
reply▲SturgeonsLaw18 hours ago
[-] That not-your-money is still going towards rewarding user-hostile decisions
reply▲No ethical consumption under capitalism. My phone has minerals in it mined under slave-like conditions.
I'm all for everyone going full Libra - we do it at my co-op - but it makes sense to me that venture funded companies would "play the game" and light investor money on fire because, first, who gives a shit, and second, the investors want you to do that anyway so they can find out as fast as possible if you're a unicorn.
At my co-op, I spend hours writing future proof code and integrating FOSS solutions that I hope will serve us forever. When I'm at a startup, I'm looking for the fastest, maybe cheapest solution. YC gave us 200k in AWS credit? Guess we're on AWS. Another company in the cohort is some LLM IDE ala cursor and gave us a year free? Sure, burn tokens their investors are paying for, more agents for me. Vercel offers us a year of free hosting? Great, I hate nextjs but Claude loves it so fuck it, we deploy a nextjs app on vercel and lock ourselves deep into that ecosystem. Our product may not look like this at all in a year so I may be rewriting it in Vue or whatever when the vercel bills start coming in. Doesn't matter.
“not my money” thinking will always indirectly lead to bad things for you on a long enough timeline.
Edit: this got flagged, but if you end up in an unprofitable position because you chose Oracle as a vendor and they squeezed your company so hard they need to choose between paying you (either via a raise or via actually employing you) or staying with a vendor thats squeezing them; they will choose the latter, as its short term cheaper.
Whatever your issues with the price, this comment is truly wild. When you’re using commercial PyCharm, you’re paying JetBrains to “run their software on your hardware.”
reply▲A one-off software license (or even a subscription) is completely different. The issue is metered billing for something you are paying for already which costs the company nothing. The equivalent is not only paying a flat monthly fee to Adobe for access to Photoshop, but also an additional charge for how long you have it open on your machine every month.
reply▲paying for the time the debugger is active would feel bad though.
reply▲raxxorraxor10 hours ago
[-] I use Gitea and think it is superior to GitHub. Can be quite well integrated in the usual Microsoft corporate environment easily as well, so you don't even need to create users. Perhaps setup two or three groups and you are done. Can be up and running in a few little hours if you start with nothing aside your domain controller.
It also doesn't randomly fail and if it would, you can probably fix it yourself.
I don't think actions on a git repository host is a good way to fix poor deployment strategies if it goes beyond pushing a package to npm and co. Just to poke at the wound again.
But Gitea has interfaces here as well, didn't try them though.
I honestly don't have any issue paying the self-hosted runner fee. Paying it and counting it against our total allocated minutes when we bought the machine is going too far though.
reply▲it's just software.
it changes and you move on.
Takes like these do not account for the value you gained by using the software in the meantime. Here are 2 scenarios:
1) company uses exclusively free software, spends more time dealing with the shortcomings of said software than developing product, product is half baked and doesn't sell well, company dies.
2) company uses proprietary but cheap/free (as in beer) software that does the job really well, focuses on developing product, product is good and sells well, company how has a ton of money they could use to replicate the proprietary product from scratch if they wanted to.
A purist approach like in scenario 1 leaves everyone poor. A pragmatic approach like scenario 2 ends up earning enough money that can be used to recreate the proprietary software from scratch (and open-source it if you wanted to).
In this case the problem isn't even the proprietariness of the software, it's the fact that companies are reliant on someone else hosting the software (GH being FOSS wouldn't actually change anything here - whoever is hosting it can still enforce whatever terms they want).
FOSS alternatives already exist, it's just that our industry is so consumed by grifters that nobody knows how to do things anymore (because it's more profitable for every individual not to); running software on a server (what used to be table stakes for any shop and junior sysadmin) is nowadays lost knowledge. Microsoft and SaaS software providers are capitalizing on this.
Your scenarios seem to hinge on OSS having lots of warts while proprietary software is perfect.
In reality you have to also make concessions with proprietary software, so the moat is not as large as your comment makes it seem imo.
embedding-shape22 hours ago
[-] > A purist approach like in scenario 1 leaves everyone poor.
That depends, not always. Sometimes the employees of said company manages to contribute back upstream, on the dime of the company. If the "free software" they used and contributed to have a lot of users, it's certainly not "leaves everyone poor" but rather "helps everyone, beyond monetary gain".
Sure, you can make the argument that it isn't that great for the company, and you may be right. But the world is bigger than companies making money, killing a few companies along the way to make small iterative steps on making free software for absolutely everyone is probably a worthwhile sacrifice, if you zoom out a bit.
Even purely from an altruist perspective I’d argue scenario 2 makes more sense as the resulting money can be used to fund a lot more open-source contributions.
reply▲Could in theory is very different from what actually happens.
In the end the purists approach results in better more productive software across even slightly longer timescales. That ultimately produces more value and thus a richer society than the kind of short term pump and dump schemes which SV is so fond of. Who captures that value is a different story than was that value created.
It could, but would it? For-profit companies usually don't suddenly turn around and start funding FOSS, unless that was part of their core mission from the start. If a company aims to make as much money, then that tends to be the mission they stick with, for better or worse.
reply▲or alternative hire right people that know what they are doing and don’t need a whole lot of junk to work on and deploy. I have been coding 31 years now and don’t have the slighest clue why anyone would ever need a “github action”
reply▲There's value in enforcing checks on the server side to avoid people accidentally/maliciously merging code that doesn't pass said checks. Checks can be linters, security scanners, etc.
reply▲why on the server?!
reply▲Because then you protect against a compromised/misbehaving developer workstation. No matter what the individual developer does, the server will prevent a PR being merged if it doesn’t pass the server-enforced checks.
Running builds on a designated server would also protect against malware on a developer’s machine silently embedding itself into the resulting artifact and then deployed to production.
franklyworks17 hours ago
[-] This was probably the question to ask before declaring it all as junk.
reply▲> Checks can be linters, security scanners, etc.
The first checks I setup are build and test. The rest is “extra”.
The first part is wrong, it’s a question of org size. A lot of large orgs hand roll a lot of these things, they call it developer excellence.
And your last paragraph hits the nail on the head, people are afraid to run their own software.
jerdthenerd13 hours ago
[-] While I agree with the spirit of your statement "people are afraid to run their own software", I feel like this assumes that people are the ones choosing the software they run. I wish my teams could run more things ourselves, but are told no by our systems and infrastructure staff.
Any self hosted service in an enterprise means that you're dealing with all the headaches that come with that including: backups, user/role creation and mapping maintenance, infrastructure scaling needs, OTEL or other monitoring, etc.
It's an easier decision for VPs to pay GitHub anything less than the man hours required to execute the above tasks because it's a "not our problem" fee.
Perhaps our industry should adopt a different approach, that fills in the gap between those.
- You host open-source software on your own hardware.
- You pay a company for setup and maintenance by the hour.
Have any suggestions to those community-developed and maintained options?
reply▲Gitea. Gitlab (ish?).
reply▲GitLab actually implemented Actions first back in the day (called CI/CD). I remember GitHub was following their lead.
reply▲Which is funny reading how TFA tries to feign ignorance:
> When we shipped Actions in 2018, we had no idea how popular it would become.
Gitea scales really badly with large repos in my experience. Gitlab works a lot better mostly because you can just throw more hardware at it. This is with a pretty large git repo and a lot of daily commits.
reply▲On the other hand, gitlab is a memory hog. You need a big vm dedicated to it.
We were on codeberg for a couple years and it was fine.
Yeah Gitlab is a pig, but that’s what I meant with you can throw hardware at the problem. I’ve been meaning to check out Codeberg for personal project hosting since it seems to address the shortcomings of gitea
reply▲brightball13 hours ago
[-] You can also just use Gitlab Cloud but setup as many self hosted runners as you like.
reply▲GitLab scales much better horizontally than it does vertically.
4x 4c/16gb instances will perform much better than one 16 core 64GB instance.
>Gitea scales really badly with large repos in my experience.
Isn't it written in this super scaling language that everybody says scales super well?
What is the problem with it?
Madmallard13 hours ago
[-] I don't understand how once these companies go down the user hostile hell-hole... like why do we allow them to keep operating?
How is there not a collective decision to dissolve them?
I deleted my account the day Microsoft acquisition was confirmed. If more people did that, maybe we wouldn't be here.
reply▲haven't logged into GitHub since they added mandatory 2FA
reply▲Same here, my commit history chart makes people think I suddenly died on October 21, 2023, going from all green to all white.
Bitbucket also implemented 2FA, but it's 100% optional, so I'm sticking with that for the moment.
> Now it is paying money for running their software on your own hardware.
(facepalm)
I got contacted by our rep a couple weeks ago, who informed me of this news. I thought it was a disaster and it really pissed me off. The rep couldn't even explain the reasoning well. It basically summed up to "because we can" and "where are you going to go?". He was shocked to find out that I didn't like it.
We currently self-host on kubernets/aws. The thing that really got to me isn't the new charge per se. It's the fact that GHA has a ton of problems. I can hold my nose and deal with them when it's free. But now that you're squeezing me, at least you could have created something like GHA 2.0 and added a charge for that. Instead, there are vague roadmap promises which don't even include things that I care about. Specifically:
- Jenkins had better kubernetes integration years ago. It's crazy that GHA can't beat that.
- "Reintroducing multi-label functionality" - yeah, so they first broke it. They did supply "reasons", which looked like they never talked to a customer. [1]
- Still no SDK of any kind.
- "Actions Data Stream" - or you can just fix your logging.
There are dozens more complains, which are easy enough to find. This kind of an approach just makes me want to make sure that I don't use GHA again. Even if I end up paying another vendor, at least I'll be treated as a customer.
[1] - https://github.com/orgs/community/discussions/160682#discuss...
Any official Github action today:
"Thank you for your interest in this GitHub action, however, right now we are not taking contributions.
We continue to focus our resources on strategic areas that help our customers be successful while making developers' lives easier. While GitHub Actions remains a key part of this vision, we are allocating resources towards other areas of Actions and are not taking contributions to this repository at this time. The GitHub public roadmap is the best place to follow along for any updates on features we’re working on and what stage they’re in."
They are focusing on an Azure migration for then next 2 years...
reply▲Nearly 20 years ago, some VP at a security products company now owned by Broadcom threatened us during contract renewal with, "The price is what it is. Your contract is up in two weeks. What are you going to do? Move to a competing product?"
We had it done with a week to spare.
Never underestimate the power of spite! :)
reply▲This kinda change also has some different gears turning in my head. At $0.002 / build-minute, some of our large software integration tests would cost us around 15 - 20 cents. Some of our ansible integration tests would be 5 - 10 cents - and we run like 50 - 100 of those per day. Some deployments might cost us a cent or two.
Apples to oranges, naturally, but like this, our infra-jenkins master would pay for itself in hosting in a week of ansible integration testing compared to what GHA would cost. Sure, maintenance is a thing, but honestly, flinging java, docker and a few other things onto a build agent isn't the time-consuming part of maintaining CI infrastructure.
And I mean sure, everything is kinda janky on Jenkins, but everything falls into an expectable corridor of jank you get used to.
> Sure, maintenance is a thing, but honestly, flinging java, docker and a few other things onto a build agent isn't the time-consuming part of maintaining CI infrastructure.
Depending on your workplace, there's a whole extra layer of bureaucracy and compliance involved if you self-host things. I aggressively avoid managing any VMs for that reason alone.
Luckily, at work we are this layer of bureaucracy and compliance. I'm very much pushing the agenda and idea that managing a stateful, mutable linux VM is a complex skill on it's own and incurs toil that's both recurring and hard to automate. The best case to handle that is to place your use case into our config management and let us manage it.
Most modern development workflows should just pickup a host with some container engine and do their work in stateless containers with some external state mapped in, like package caches. It's much easier for both sides in a majority of cases.
Seattle350313 hours ago
[-] > And I mean sure, everything is kinda janky on Jenkins, but everything falls into an expectable corridor of jank you get used to.
This is kinda where I am. No one really feels like they are selling a premium "just works" product. Its all jank. So why it the jank I chose at the price I chose?
At the moment I'm self hosting gitlab runners. Its jank. But it's free.
A while ago I set out to find a replacement for Jenkins. On prem with a comparable feature set. What I found out is that Jenkins is the worst apart from all the others.
reply▲> And I mean sure, everything is kinda janky on Jenkins, but everything falls into an expectable corridor of jank you get used to.
Self-hosting Jenkins on an EC2 instance is probably going to result in a _better_ experience at this point. Github Cache is barely better than just downloading assets directly, and with Jenkins you can trivially use a local disk for caching.
Or if you're feeling fancy and want more isolation, host a local RustFS installation and use S3 caching in your favorite tools.
Self-hosting on a host whose data actually persists is an even better experience, as it removes a lot of the tedium and workarounds such as extracting/down-/up-loading caches and so on. Get another host for redundancy and call it a day.
Hardware is getting cheaper and cheaper, but the fear-mongering around running a Linux machine has successfully prevented most businesses from reaping those cost reductions.
wereHamster8 hours ago
[-] I repurposed old M1/M4 Mac Mini's at my workplace into GitHub action runners. Works like a charm, and made our workflows simpler and faster. Persisting the working directory between runs was a big performance boost.
reply▲> Hardware is getting cheaper and cheaper
Unfortunately not anymore and not in the foreseen future if we don't see some AI investment corrections.
Complete persistence has its downsides, as you can start getting "path dependency". E.g. a build succeeds only because some images were pre-cached by a previous build.
But having an _option_ to not download everything every time is great. You can add a periodic cache flushing, after all.
Last place I worked had long running end to end tests that would take 30 minutes on GHA (compared to maybe 5 locally) on every PR. This is going to make that a very expensive endeavour
reply▲We host a fair bit of Terraform code in a repos on GitHub, including the project that bootstraps and manages our GH org’s config: permissions, repos, etc.
Hilariously, the official Terraform provider for GitHub is full of N+1 API call patterns — aka exponential scaling hotspots — so even generating a plan requires making a separate (remote, rate-limited) API call to check things like the branch protection status of every “main” branch, every action and PR policy, etc. As of today it takes roughly 30 minutes to do a full plan, which has to run as part of CI to make sure the pushed TF code is valid.
With this change, we’ll be paying once to host our projects and again for the privilege of running our own code on our own machines when we push changes…and the bill will continue to grow exponentially b/c the speed of their API serves to set an artificial lower bound on the runtime of our basic tests.
(To be fair, “slow” and “Terraform” often show up and leave parties at suspiciously similar times, and GitHub is far from the only SaaS vendor whose revenue goes up when their systems get slower.)
madeofpalk18 hours ago
[-] It seems clear to me this is in response to all the third party GHA runners who were undercutting GitHub by just reselling cloud instances for cheaper.
They’ve lowered their runner costs to compete, and introduce minimum charge to discourage abd make sure they still get paid.
100% this.
They could have made their service better or more competitive like with price but instead they chose this route. SMH.
And if you want any concurrency at all, you need 1 runner registration per concurrent job. And each runner needs its own user. And each runner requires a full and separate copy of the runner software, which is large (hundreds of megs) and self-updates.
reply▲paulddraper22 hours ago
[-] You don't need your own user.
The rest is correct. (Though you can hardlink the installation.) And you can disable self-update, though it does it by default.
Ah right, I've forgotten because I'm using a multi-user strategy and a patched version of the runner at this point anyway. The config directory for each runner is normally based on its install path (insane), something like that?
reply▲Hard-linking and running concurrent self-updates, sounds like a recipe for disaster.
reply▲Yet it works perfectly which forgejo/gitea (uses the act runner)
reply▲And they are even on their third (fourth?) from-scratch rewrite of their agent server. When will they get it right? (Rhetorical question)
reply▲I am developing a self-hosted solution for this [1]. It’s true that it’s somewhat of a pain but JIT runners allow a lot of flexibility that we don’t find elsewhere.
[1] https://runs-on.com
In my experience even with runs-on it is a bit painful with runner cancellations.
reply▲Introducing a separate charge specifically targeting those of your customers who choose to self-host your hilariously fragile infrastructure is certainly a choice.. And one I assume is in no way tied to adoption/usage-based KPIs.
Of course, if you can just fence in your competition and charge admission, it'd be silly to invest time in building a superior product.
We've self-hosted github actions in the past, and self-hosting it doesn't help all that much with the fragile part. For github it is just as much triggering the actions as it is running them. ;) I hope the product gets some investment, because it has been unstable for such a long time, that on the inside it must be just the usual right now. GitHub has by far the worst uptime of any SaaS tools we use at the moment, and it isn't even close.
> Actions is down again, call Brent so he can fix it again...
We self host the runners in our infrastructure and the builds are over 10x faster than relying on their cloud runners. It's crazy the performance you get from running runners on your own hardware instead of their shared CPU. Literally from ~8m build time on Gradle + Docker down to mere 15s of Gradle + Docker on self hosted CPUs.
reply▲This! We went from 20!! minutes and 1.2k monthly spend on very, very brittle action runs to a full CI run in 4 minutes, always passing, by just by going to Hetzner's server auction page and bid on a 100 euro Ryzen machine.
reply▲After self hosting our builds ended up so fast, that we were actually waiting for was GitHub scheduling our agents, rather than it being the job running. It sucked a bit, because we'd optimized it so much, but we on 90th percentile saw that it took 20-30 seconds for github to schedule the jobs as they should. Measured from when the commit hit the branch, to the webhook begin sent.
reply▲My company uses GitHub, GitLab, and Jenkins. We'll soon™ be migrating off of GitLab in favor of GitHub because it's a Microsoft shop and we get some kind of discount on GitHub for spending so much other money with Microsoft.
Scheduling jobs, actually getting them running, is virtually instant with GitLab but it's slow AF for GitHub for no discernable reason.
lmao I just realized on this forum writing this way might sound like I own something. To be clear, I don't own shit. I typically write "my employer", and should have here.
reply▲> call Brent so he can fix it again
Not sure if a Phoenix Project reference, but if it is, it's certainly in keeping with Github being as fragile as the company in the book!
It is xD On the outside it feels like a product held together with duct tape, wood glue and prayers.
reply▲Hey, don't insult wood glue like that.
reply▲Indeed, wood glue is amazing. Such slander is totally uncalled for.
reply▲steve_adams_8621 hours ago
[-] I don't know, maybe it's a compliment. Wood glue can form bonds stronger than the material it's bonding. So, the wood glue in this case is better than the service it's holding together :)
reply▲I tend to just rely on the platform installers, then write my own process scripts to handle the work beyond the runners. Lets me exercise most of the process without having to (re)run the ci/cd processes over and over, which can be cumbersome, and a pain when they do break.
The only self-hosted runners I've used have been for internalized deployments separate from the build or (pre)test processes.
Aside: I've come to rely on Deno heavily for a lot of my scripting needs since it lets me reference repository modules directly and not require a build/install step head of time... just write TypeScript and run.
We choose github actions because it was tied directly to github providing the best pull-request experience etc. We actually didn't really use github actions templating as we'd got our own stuff for that, so the only thing github actions actually had to do was start, run a few light jobs as the CI was technically run elsewhere and then report the final status.
When you've got many 100s of services managing these in actions yaml itself is no bueno. As you mentioned having the option to actually be able to run the CI/CD yourself is a must. Having to wait 5 minutes plus many commits just to test an action drains you very fast.
Granted we did end up making the CI so fast (~ 1 minute with dependency cache, ~4 minutes without), that we saw devs running their setup less and less on their personal workstations for development. Except when github actions went down... ;) We used Jenkins self-hosted before and it was far more stable, but a pain to maintain and understand.
This is absolutely bananas; for my own CI workflow I'll have to pay $140+/month now just to run my own hardware.
reply▲Am I right in assuming it’s not the amount of payment but the transition from $0 to paying a bill at all?
I’m definitely sure it’s saving me more than $140 a month to have CI/CD running and I’m also sure I’d never break even on the opportunity cost of having someone write or set one up internally if someone else’s works - and this is the key - just as well.
But investment in CI/CD is investing in future velocity. The hours invested are paid for by hours saved. So if the outcome is brittle and requires oversight that savings drops or disappears.
Have you ever set up GitHub Actions? The outcome is brittle because of their platform, not because of my inability to do CI.
reply▲Exactly this. I've used Jenkins, Travis, CircleCI and all of them were so easy in comparison to the github actions runner mess.
reply▲I use them minimally and haven't stared at enough failures yet to see the patterns. Generally speaking my MO is to remove at least half of the moving parts of any CI/CD system I encounter and I've gone a multiple of that several times.
When CI and CD stop being flat and straightforward, they lose their power to make devs clean up their own messes. And that's one of the most important qualities of CI.
Most of your build should be under version control and I don't mean checked in yaml files to drive a CI tool.
featherless23 hours ago
[-] This is not investment in CI/CD. I already did that, by buying and investing in my own hardware, my own workflows, my own caching solution.
This is like if Dropbox started charging you money for the files you have stored on your backup hard drives.
Don’t give them any ideas! This is actually a standard enshittification.
reply▲You're sounding a lot like a Microsoft damage control artist.
reply▲The only company I’ve held a grudge against longer than MS is McDonalds and they are sort of cut from the same cloth.
I’m also someone who paid for JetBrains when everyone still thought it wasn’t worth money to pay for a code editor. Though I guess that’s again now. And everyone is using an MS product instead.
I'm curious, what are you doing that has over 1000 hours a month of action runtime?
reply▲I run a local Valhalla build cluster to power the
https://sidecar.clutch.engineering routing engine. The cluster runs daily and takes a significant amount of wall-clock time to build the entire planet. That's about 50% of my CI time; the other 50% is presubmits + App Store builds for Sidecar + CANStudio / ELMCheck.
Using GitHub actions to coordinate the Valhalla builds was a nice-to-have, but this is a deal-breaker for my pull request workflows.
Cool, that looks a lot nicer than the OBD scanner app I've been using.
reply▲reply▲I found that implementing a local cache on the runners has been helpful. Ingress/egress on local network is hella slow, especially when each build has ~10-20GB of artifacts to manage.
reply▲What do you use for the local cache?
reply▲ZeroFS looks really good. I know a bit about this design space but hadn't run across ZeroFS yet. Do you do testing of the error recovery behavior (connectivity etc)?
reply▲This has been mostly manual testing for now.
ZeroFS currently lacks automatic fault injection and proper crash tests, and it’s an area I plan to focus on.
SlateDB, the lower layer, already does DST as well as fault injection though.
Wow, that's a very cool project.
reply▲1 hour build/test time, 20 devs, that's 50 runs a month. Totally possible.
reply▲GH actions templates don’t build all branches by default. I guess it’s due to them not wanting the free tier to use to much resources. But I consider it an anti-pattern to not build everything at each push.
reply▲sunnyday_00210 hours ago
[-] That is because GH Actions is event based, that is more powerful and flexible than just triggering on every push and not letting it be configured.
```
on: push
```
is the event trigger to act on every push.
You'll waste a lot of CI on building everything in my opinion, I only really care about the pull request.
I resorted to a local forgejo + woodpecker-ci. Every time I am forced back to GitHub for some reason it confirms I made the right choice.
In my experience gitlab always felt clunky and overly complicated on the back end, but for my needs local forgejo is better than the cloud options.
They still host all artefacts and logs for these self-hosted runs. Probably costs them a fair bit
reply▲They already charge for this separately (at least storage). Some compute cost may be justified but you'd wish that this change would come with some commitment of fixing bugs (many open for years) in their CI platform -- as opposed to investing all their resources in a (mostly inferior) LLM agent (copilot).
reply▲Copilot uses other models, not (necessarily?) its own, so I’m not sure what you mean.
reply▲It does leverage various models, but
- github copilot PR reviews are subpar compared to what I've seen from other services: at least for our PRs they tend to be mostly an (expensive) grammar/spell-check
- given that it's github native you'd wish for a good integration with the platform but then when your org is behind a (github) IP whitelist things seem to break often
- network firewall for the agent doesn't seem to work properly
raised tickets for all these but given how well it works when it does, I might as well just migrate to another service
There's absolutely no way that the cost scales with the usage of my own hardware. I cannot fathom this change in any way or form. Livid.
reply▲Meanwhile I'm just running `pytest`, `pyproject-build`, `twine` etc. at the command line....
(People seem to object to this comment. I genuinely do not understand why.)
It passes on my machine. YOLO!
reply▲You don't trust devs to run things, to have git hooks installed, to have a clean environment, to not have uncommitted changes, to not have a diverging environment on their laptop.
Actions let you test things in multiple environments, to test them with credentials against resources devs don't have access to, to do additional things like deploys, managing version numbers, on and on
With CI, especially pull requests, you can leave longer running tests for github to take care of verifying. You can run periodic tests once a day like an hour long smoke test.
CI is guard rails against common failure modes which turn requiring everyone to follow an evolving script into something automatic nobody needs to think about much
> You don't trust devs to run things, to have git hooks installed, to have a clean environment, to not have uncommitted changes, to not have a diverging environment on their laptop.
... Is nobody in charge on the team?
Or is it not enough that devs adhere to a coding standard, work to APIs etc. but you expect them to follow a common process to get there (as opposed to what makes them individually most productive)?
> You can run periodic tests once a day like an hour long smoke test.
Which is great if you have multiple people expected to contribute on any given day. Quite a bit of development on GitHub, and in general, is not so... corporate.
I don't deny there are use cases for this sort of thing. But people on HN talking about "hosting things locally" seem to describe a culture utterly foreign to me. I don't, for example, use multiple computers throughout the house that I want to "sync". (I don't even use a smartphone.) I really feel strongly that most people in tech would be better served by questioning the existing complexity of their lives (and setups), than by questioning what they're missing out on.
I think you could learn a lot about the other use cases if you asked some genuine questions and listened with intent
reply▲colechristensen18 hours ago
[-] It seems like you may not have much experience working in groups of people.
>... Is nobody in charge on the team?
This isn't how things work. You save your "you MUST do these things" for special rare instructions. A complex series of checks for code format/lint/various tests... well that can all be automated away.
And you just don't get large groups of people all following the same series of steps several times a day, particularly when the steps change over time. It doesn't matter how "in charge" anybody is, neither the workplace nor an open source project are army boot camp. You won't get compliance and trying to enforce it will make everybody hate you and turn you into an asshole.
Automation makes our lives simpler and higher quality, particularly CI checks. They're such an easy win.
Because you appear completely oblivious and deliberately naive about the entire purpose of CI.
reply▲Based on my experience I really do think most people are using it for things that they could perfectly well do locally with far less complication.
Perhaps that isn't most use of it; the big projects are really big.
Care to provide examples?
Fundamentally, yes, what you run in a CI pipeline can run locally.
That's doesn't mean it should.
Because if we follow this line of thought, then datacenters are useless. Most people could perfectly host their services locally.
> Because if we follow this line of thought, then datacenters are useless. Most people could perfectly host their services locally.
There are a rather lot of people who do argue that? Like, I actually agree that non-local CI is useful, but this is a poor argument for it.
I'm aware of people arguing for self-hosting some services for personal use.
I'm not aware of people arguing for self-hosting team or enterprise services.
eudamoniac20 hours ago
[-] Well, they are. Selling the team or enterprise a license to do just that is a rather large part of many businesses.
reply▲Runners aren’t fragile, workflows are.
The runner software they provide is solid and I’ve never had an issue with it after administering self-hosted GitHub actions runners for 4 years. 100s of thousands of runners have taken jobs, done the work, destroyed themselves, and been replaced with clean runners, all without a single issue with the runners themselves.
Workflows on the other hand, they have problems. The whole design is a bit silly
it's not the runners, it's the orchestration service that's the problem
been working to move all our workflows to self hosted, on demand ephemeral runners. was severely delayed to find out how slipshod the Actions Runner Service was, and had to redesign to handle out-of-order or plain missing webhook events. jobs would start running before a workflow_job event would be delivered
we've got it now that we can detect a GitHub Actions outage and let them know by opening a support ticket, before the status page updates
> before the status page updates
That’s not hard, the status page is updated manually, and they wait for support tickets to confirm an issue before they update the status page. (Users are a far better monitoring service than any automated product.)
Webhook deliveries do suffer sometimes, which sucks, but that’s not the fault of the Actions orchestration.
I'm seeing wonky webhook deliveries for Actions service events, like dropping them completely, while other webhooks work just fine. I struggle to see what else could be responsible for that behaviour. it has to be the case that the Actions service emits events that trigger webhook deliveries & sometimes it messes them up.
reply▲gheltlkckfn16 hours ago
[-] The orchestration service has been rewritten from scratch multiple times, in different languages even. How anyone can get it this wrong is beyond me.
The one for azure devops is even worse though, pathetic.
> We are introducing a $0.002 per-minute Actions cloud platform charge for all Actions workflows across GitHub-hosted and self-hosted runners.
Charging for self-hosted runners is an interesting choice. That's the same cost as their smallest hosted runners [1]
[1] - https://docs.github.com/en/billing/reference/actions-runner-...
Pushing you towards their hosted runners which will show up in their Azure usage numbers and drive the stock price
reply▲Ah yes, vertical integration and oligopoly.
Really Dianne?
it'd be great if they can couple this with an SLA for GitHub actions so we won't have to end up paying as much..
(ofc, that'd only mean they stop updating the status page, so eh)
For what it's worth, they already fail to update the status page. We had an "outage" just this morning where jobs were waiting 10+ minutes for an available runner -- resolved after half an hour or so but nothing was ever posted
https://downdetector.com/status/github/
Last week (Sunday to Sunday) I had a repo running a lot of cron workflows 24/7. After like 4 or 5 days I exceeded the free limits (Pro plan) and so set up self hosted runners.
After like day 2 my workflows would take 10-15 minutes past their trigger time to show up and be queued. And switching to the self hosted runners didn't change that. Happens every time with every workflow, whether the workflow takes 10 seconds or 10 minutes.
I don't want to shit on the Code to Cloud team but they act a lot like an internal infrastructure team when they're a product team with paying customers
reply▲It also seems odd that there's no discussion of using webhooks to replace the self-hosted runners for free, given that it would basically be working the same way. The only difference is who commits get attributed to, and where you go for logs and artifacts (these can probably ride along draft/prereleases or something)
reply▲Yep - Bitbucket made a similar move recently and I guess they are just following along. I'd love to get the justification of that fee tho…
Edit: Confused GitLab and Bitbucket
> justification of that fee
ZIRP ended, its remaining monopoly money has been burnt through, and the projected economy is looking bleak. We're now in the phase where everything that can be monetized is being monetized in every way that can be managed.
Free tiers evaporate. Fees appear everywhere. Ads appear everywhere, even where it was implied they wouldn't. The lemons must be squeezed.
And because everybody of relevance is in that mode, there's little competitive pressure to provide a specific rationale for a specific scheme. For the next few years, that's all the justification that there needs to be.
Your edit made your post confusing for us now...
I thought that "Bitbucket" was in your original post and you added only your edit message to say that it was, in fact, Gitlab and not Bitbucket that added cost for self-hosted runners.
gheltlkckfn16 hours ago
[-] Actually, Atlassian is just getting rid of their on-prem hosted software all together. It’s not a product they will offer any longer.
reply▲I initially felt a bit offended when I saw this. Then I thought about it and at the end of the day there's a decent amount of infrastructure that goes into displaying the build information, updating it, scanning for secrets and redacting, etc.
I don't know if it's worth the amount they are targeting, but it's definitely not zero either.
You would think the fat monthly per-seat license fee we also pay would be enough to cover the costs of checks notes reading some data from the DB and hosting JSON APIs and webpages.
reply▲Yeah, I think we’re seeing some fallout from how much developer infrastructure was built out during the era where VCs were subsidizing everything, similar to how a lot of younger people complained about delivery charges going up when they had to pay the full cost. Unfortunately, now a lot of the competition is gone so there isn’t much room to negotiate or try alternate pricing models.
reply▲Does it make sense to accept charge per minute when you are hosting yourself? When GHA is not very good?
reply▲That's a surprise, do you have a link to their announcement?
reply▲reply▲Thanks for clarifying it! I left bitbucket many years ago when they changed their UI to a new style that was awful to use...
reply▲gheltlkckfn16 hours ago
[-] Rugpull 101. It’s how you make money in the current economy.
reply▲reply▲Starting an external CI company for GitHub is becoming more interesting now.
Gitlab offers ability to do CI for external repositories. Travis CI was what everyone used before Github Actions. Time for a new Travis?
reply▲There are a few like buildkite
reply▲Buildkite is so dope; love them
reply▲Their website is terrible though. Weird geeky interface, and I could only find reams and reams of gushing copy about how great they are. Nothing concrete about
why.
Also quite expensive!
CI is one of those twilight zone things that by the time you need something like buildkite, you're making a lot of money, otherwise why would you have such a complicated CI setup? To do it right, you basically need to start spending buildkite money either way in staffing or buying buildkite. There are probably under 50k organizations in the world that need something like buildkite.
It does have a big 'it shouldn't be this expensive' energy, but the market has shown it needs to be unfortunately. Nobody really survives in the CI world without going to complete neglect mode or goes expensive like buildkite I've found. It reminds me a lot of home automation / IoT. Lutron costs almost $100 a light switch for really silly economic reasons unfortunately even though the tech is basically unchanged since the 90s.
The interface is also geeky because the only people who are going to even realize you need to spend money on this are other software professionals.
Someone123422 hours ago
[-] I really enjoy how they list the price PER MINUTE to make it sound like this isn't absurdly expensive. A lot of people leave their self-hosted runners running 24/7 because, after all, they're self-hosted.
This is $2.88/day, $86.4/month, $1051.2/year. For them to do essentially nothing.
Most notably, this is the same price as their hosted "Linux 1-core" on a per-minute basis. Meaning they're charging you the same for running it yourself, as you'd pay for them to host it for you...
> For them to do essentially nothing.
Orchestration, logging, caching, result storage.
It's not nothing. Whether it's worth it to you is a value judgement, and having run a bunch of different CI systems I'd say this is still at least competitive.
PunchyHamster21 hours ago
[-] They are charging for storage separately already! Why are you lying ?
reply▲I know they charge for Artifact storage, but outside of uploaded artifacts I don't think that the logs and results of builds are billed separately?
Additionally, I thought that caching came out of a separate limit, and was not billed like artifact storage?
Lying implies intent, I don't think the person you're replying to is necessarily lying, even though they might be wrong on this specific point.
reply▲gheltlkckfn16 hours ago
[-] GH enterprise cloud is charging for storage separately, as an organization admin just navigate to the org admin page to see it.
reply▲How can they charge for something self hosted per minute? Thats very weird to me. If I run the software I should pay a single time only, if I don't own it then why self-host im the first place?
Maybe this is designed to scare people away from self-hosting altogether?
Someone123421 hours ago
[-] I do believe, this is to disincentivize self-hosting for smaller-medium workloads. In essence, they're saying that if you're small, you should just use their Linux 1-Core, but if you're medium-to-large you won't care about the high cost.
It is a way of increasing lock-in for smaller-medium clients, without driving away their medium-large ones.
Wait.. is this how they're billing it?? Not the duration of runs??
reply▲Factor117722 hours ago
[-] It is duration of runs. He was just highlighting the absurde cost if you were to run it 24/7 like some people with their own self hosted runners do.
reply▲I am not understanding something.
If its the price of runs, then its not always running.
If its price of the agent to exist, then thats not paying per runs- then you’re right that people tend to leave their runners online 24/7- but I’ve never worked anywhere that had workers building 24/7.
OP means to say he has many jobs in the merge queue that the runners are always busy 24/7.
This is not uncommon in some orgs - less number of concurrent runners, slow builds, loads of jobs because of automation or how hooks for the runners are setup.
In the context of discussion that doesn't matter, OP's point distills to that they use minimum of 720 hours / month of orchestration time or some multiple of that on self hosted runners running 24x7.
Github will now charge $84 extra per month for single self-hosted runner running 24x7 - i.e. that is the cost for 43,200 build minutes for only their orchestration alone.
In a more typical setup that is equivalent to say 5 self-hosted running running ~4.5 hours a day(i.e 144/hours/runner/month)
If you have a lot of not very time sensitive jobs, e.g. large merge trains, it was reasonable to have a not very fast runner run close to full utilization. Now that you'd pay by the run-minute, it'll be cheaper to move to a faster runner and run it at 10%.
reply▲OP replied and clarified that that's not the case.
reply▲His workload is close to the more typical one i mentioned as scenario b. It will cost them $84/month.
For me, we do about 800,000 build minutes/month, for orchestration alone it is going to be $1600/month. In contrast the runner host we use (Namespace labs) cost $0.0015 / minute[1] which is less than orchestration cost for GH, that is just ridiculous.
---
[1] It is even worse, the first 250,000 minutes is fixed at $250, so the base is $0.001 /minute for the runner.
I guess some people just always have something running since it's owned hardware. Daily builds of popular OSS projects or constant vuln scans or whatever?
reply▲Someone123420 hours ago
[-] When you've already paid for the hardware, it is essentially free after that (aside electricity, I suppose). So there wasn't a reason to ration our runners, and we actually added additional workloads/scans/etc just because we could.
reply▲Someone123420 hours ago
[-] We're targeting 4x different deployment pipelines, so while we aren't running 24/7, we are running the same number of hours but split over all our runners. Often runs are queued during our busy 8-hour work-day, and then unused for 16-hours.
Either way, we will likely pay 8-hours4-pipelines5-days=160 hours per week, just shy of 168-hours for true 24/7. This currently costs $0 just for context.
PunchyHamster21 hours ago
[-] You can get far bigger VM for that per month. It's ridiculus.
Of course entirely expected after MS buyout, if anything I'm surprised it took that long
Yup. Took wayyy longer than I actually expected as well. But the change of top management and closer integration with the whole MS behemoth is likely to make those kind of things accelerate now
reply▲$1k per year if you run an action 24/7. How many minutes per month do you actually use? How does that compare to the cost of the machines being used as runners?
The real mistake was GH not charging anything for self-hosted runners in the first place, setting an expectation.
> A lot of people leave their self-hosted runners running 24/7
Don't they generally only kick in when you push or merge?
I was about to call you off and say your math is wrong, you must be an order of magnitude wrong.
But you are right, this is ridiculous!
Per machine. Definitely more than one machine here.
reply▲That's not a move of a company that thinks it can still grow. That's a Netflix "we have 90% of the market, let's squeeze them" move.
This is the beginning. We have all seen this pattern over the last 5+ years. You know their next few moves.
reply▲The Netflix one worked
reply▲Worked for them not for the consumers
reply▲short-term maybe, but piracy is making a come back
reply▲It never left
reply▲array_key_first23 hours ago
[-] It actually kind of did for a lot of people. Streaming was cheap, available, and convenient.
Now it's none of those three. Once again, choosing not to pirate is just an objectively wrong choice. It's a worse experience, with worse quality, worse availability, and at a higher price tag.
> Choosing not to pirate is just an objectively wrong choice. It's a worse experience, with worse quality, worse availability, and at a higher price tag.
Choosing not to pirate and not to consume simultaneously is not necessarily a wrong choice. A difficult one? Yes. But I propose that it could be beneficial for your mental (and maybe physical) health.
array_key_first16 hours ago
[-] This is the approach I took with most things, so you're right. But still, TV can be some of the highest quality and engaging media you can find. I mean, it's not short form slop or thinly veiled advertisment... If you look in the right places.
reply▲I went almost 20 years without sailing the high seas. It was the death of DVD Netflix that really did it for me.
With DVD, Netflix if something I wanted to watch wasn't on any of my streaming services, it was almost guaranteed to be on DVD Netflix. That fallback doesn't exist anymore.
estimator729216 hours ago
[-] Yeah, once I grew up and started making money, I quit pirating. Just didn't have a need for it anymore.
But when streaming started to really go down the toilet I already had a homelab so I spun up radarr and Jellyfin behind seven proxies for family-scale piracy. It's wonderful. This is a new golden age for piracy.
Netflix is looking out for Netflix shareholders, not for consumers, like any other public company.
reply▲Perfectly good excuse to make society worse for people. Oh wait, what's that? It's not a good excuse? Oh okay.
reply▲Nobody needs a Netflix subscription. You can just stop paying
reply▲> Perfectly good excuse to make society worse for people
What an incredibly silly accusation to make of a company/service that streams movies and television. Like you understand it is possible to dilute the concept of civic responsibility right?
?
Companies don't care about society, unless it affects profit. Companies are not people, they are cold machines that through different means try to reach the same purpose, make more money.
No one should anthropomorphize companies. They might look like they have human qualities, same way like the T800 in the Terminator looked human.
It worked to push me to unsubscribe.
reply▲rssoconnor18 hours ago
[-] I also unsubscribed, but even with you and me out, from what I read, it was a profitable move on Netflix's part. I guess I can't fault them.
reply▲The first stages always do, that's why corporations keep pulling the enshittification lever.
reply▲> Self-hosted runners: You will be charged for using the GitHub Actions cloud platform from March 1, 2026
The GitHub encrapification finally affects me. I am militantly unwilling to pay per minute to use my own computer. Time to leave. I can trigger and monitor builds myself thank you very much.
I migrated to forgejo a few years ago and never looked back. While there are some edge cases and known issues. All of my actions "just worked".
reply▲Getting acquired by Microsoft is a death sentence for any product.
The only variable is how long after acquisition before they gut it. It's almost never right away. GitHub was acquired 7 years ago, but it started showing symptoms perhaps 2 years ago.
With this I think it's clear the wound was fatal. GitHub will stumble on for a few more years with ever-decreasing quality, before going the way of Skype.
So, I guess we're all migrating to gitlab? Or is it time to launch gittube? Githamster?
The exodus from GitHub has not begun, as far as I can tell.
They seem to care much less about free users than in the past but businesses still flock to it. GitLab is the only other platform I’ve seen in the workplace of anywhere I worked, with the exception of a big tech company I worked at. They had both GitHub enterprise and an internally maintained platform which was being phased out. if I recall correctly it based on Phabricator
I've seen a small but increasing trickle of open source projects leaving github for free(libre) alternatives lately. It's not a stampede but I'd say the exodus is on its way.
The considerations for commercial users leaving github are probably pretty different, so perhaps they'll stay.
Github, i.e. MSFT will of course adjust their offer if they would start seeing an significant exodus. They could afford being a loss leader indefinitely.
There’s also a large bunch of projects that will never leave, some of which ms have influence over. I mean, some high profile open source projects hasn’t even left source forge yet, even though it has been malware infested shithole for decades.
In case you're considering moving to GitLab we currently have no plans that I'm aware of to pay from bringing your own runners. Happy to answer any questions.
reply▲Why are the self-hosted plans so expensive? Such a high bar to go from free to paid.
reply▲reply▲This looks really nice actually! European non-profit is hitting some really good buttons for me! Thanks for the tip, I hadn't heard of codeberg before!
reply▲If Microsoft had not acquired GitHub, there would not be GitHub Actions. GitHub Actions is a mediocre knock-off of Azure Pipelines, and it was launched after the acquisition.
reply▲Azure Pipeline wasn't mediocre?
reply▲This pricing model continues to incentivize them not fixing the hundreds of clearly documented issues that causes CI to be incredibly slow. Everything from their self-inflicted bottlenecking of file transfers to the safe_sleep bug that randomly makes a runner run forever until it times out.
reply▲reply▲Hopefully something ForgeFed-powered, so that we can all re-decentralize, as is right and proper.
reply▲I'm running forgejo on my NAS, including CI runners etc. Harder to share with folks but great for my personal projects (except building an iOS app, which someday I'll set a Mac Mini up for probably)
reply▲johannes123432123 hours ago
[-] > The only variable is how long after acquisition before they gut it.
Considering that the lifetime of our sun system is finite that statement is undeniably true.
Also we don't know how a non-Microsoft GitHub could have developed.
This is my first comment on HN despite being a user for over a decade -- this is one of the most outrageous pricing changes I've encountered - I couldn't believe it when I read the email earlier (I run self-hosted runners).
Anyone using GitLab or any other VCM that you'd recommend? I'm absolutely done with Github. Or is everything else just as bad?
I'm pretty happy with codeberg.org as a free host.
Alternatively, Forgejo, Gitea, or (based on praise I've seen from other people) maybe sourcehut.org.
I find GitLab's interface intolerable. Heavy reliance on javascript even for read-only access, nonintuitive organization, common operations hidden behind menus, mystifying icons... Every time I seek out a project's home and discover a GitLab instance, I find myself pausing to reconsider whether contributing to the project will really be rewarding enough to outweigh the unpleasant experience I'm about to have.
What does VCM stand for?
Gitlab interface is busy, yeah. But you it packs a lot of functionality in. If you want, you disable features like wiki and snippets to free up space on the side bar of a project. Or just look past it and find the part you want, issues merge requests, whatever.
reply▲After working for years with GitLab professionally, you know exactly where everything is.
Particularly making a contribution should anyhow be trivial - you push the branch and it shows a banner in the repo asking if you want to open a MR for the recently pushed branch.
I don't know why anyone would use GitHub actions. They seem like a weird, less powerful version of the GitLab CI. Now they want to charge for runtime on your own runner.
We self host GitLab and it’s been amazing. Never down, all the features we need. And the CI is, at least for me, easier to understand than GH actions. You’re just running scripts in a container no weird abstractions.
reply▲codeberg.org for open source, because it's a non-profit, with what it seems, very well intentioned people, with a good governance structure, and it's starting to support federation.
For a company, I'd recommend self-hosting forgejo (which also has actions), which powers codeberg.
(forgejo started as a fork of gitea)
Gitea is almost fully compatible with the GitHub runners. You might need to do small changes in the workflows.
reply▲Gitea!
And the best (maybe?) part in your case is that the CI is based on GH Actions, so you can probably reuse your workflows without the need to adapt them.
Have been using Gitlab happily for a decade, though I have never had to directly worry about the cost for larger teams.
reply▲Self-hosted gitlab here. Love it, and gitlab CI is excellent as well. Almost all product development revolves around some crappy AI integration that we don't use, and it worries me to see so much focus there instead of the core product, but the core product is still excellent.
reply▲I will try Gitea -- thanks everyone for the recommendations
reply▲Gitea has a fully compatible system AFAIK.
reply▲"Outrageous"? It's two tenths of a cent per minute. How much will it impact you, really, particularly relative to the cost of compute?
reply▲array_key_first23 hours ago
[-] It's outrageous because self hosted runners are... self hosted.
And the entire solution just sucks. It's bad, bad software. It barely works a lot of the time.
The only reason they're doing this is because they can. Which is usually a really stupid reason. And here, it is. So, that's outrageous.
> because self hosted runners are... self hosted.
Sure, but there's a separate mechanism that you need to make it all work: the orchestration. Without that, you have only the capacity to run jobs -- it's potential energy, if you will, not doing real work.
That orchestrator thus provides real value. And it's not like it's free for them to build, operate, and maintain.
The CI pipeline using my own runner is absolutely something that I expect to come for free with hosting the repository.
reply▲loginatnine22 hours ago
[-] At our company, it's ~35k USD increase annually. This is not negligeable.
reply▲That’s 2.5x what my actual runners cost. For every $100 in compute, I will be paying $250 to github. They can fuck right off
reply▲I'm curious - where are you finding runners for $0.0008/minute? What are their specs?
reply▲Ubicloud. Also faster than the analog runners github provide. Only problem is startup time is slower
reply▲>
In the past, our customers have asked us how GitHub views third-party runners long-term. The platform fee largely answers that: GitHub now monetizes Actions usage regardless of where jobs run, aligning third-party runners like Blacksmith as ecosystem partners rather than workarounds.It does? I feel like it implies that they want third-party runners like Blacksmith out of the ecosystem, which is why they're now financially penalizing customers who use them.
With these changes, three things hold:
1. Services like blacksmith and WarpBuild (I'm the founder) are still cheaper than GitHub hosted runners, even after including the $0.002/min self-hosting tax.
2. The biggest lever for controlling costs now is reducing the number of minutes used in CI. Given how slow Github's runners are, or even the ones on AWS compared to our baremetal processor single core performance + nvme disks, it makes even more sense to use WarpBuild. This actually makes a better case for moving from slow AWS instances running with actions-runner-controller etc. to WarpBuild!
3. Messaging this to most users is harder since the first reaction is that Github options make more sense. After some rational thought, it is the opposite.
Overall - it is worse for Github users, but options like blacksmith and WarpBuild are still the better option.
"WarpBuild are still the better option."
what makes you think they won't hike the control plane price again? They can turn this knob arbitrarily to put you out of business.
The statement regarding the better option is as it stands today and does not account for all possible futures.
Reg. hiking it again, they'd have to either be extremely anti-competitive and selectively apply the pricing OR apply the hike uniformly by about double the current value to match our pricing while making it completely unviable for any large co to use self-hosted github actions in the first place.
I checked the WarpBuild website and got excited because the header in the menu says you have macOS Intel runners, but then you click through and it doesn't seem to be so?
Right now at my company our biggest complaint are macOS Intel runners from GitHub which somehow take 15+ minutes to provision and are the slowest of the bunch.
joshstrange22 hours ago
[-] I can assure you WarpBuild has Mac runners that work very well. When I first switched GH only offered 1 Mac runner and it was horribly slow. Literally cut my build times in half by changing 1 line in my workflow file to use the WB runner.
Nowadays GH has more sizes by WB continues to beat them in price and performance.
It’s highway robbery what GH charges for the crap they provide. I can highly recommend WarpBuild for Mac (and Linux) runners.
I was talking specifically of macOS Intel runners. The sibling comment from the founder confirmed they don't have them.
reply▲We only have macos arm64 (M-series) runners. Can you point me to the intel reference so I can fix it?
reply▲mattraibert16 hours ago
[-] Hover the top nav. Under "CI Runners" it's says:
macOS Runners
Apple Silicon and Intel support
fixed it - sorry about that.
reply▲That's clearly the case, this is a three-pronged manoeuver :
- Introducing a cheap 1-core runner
- Lowering the price of GitHub-hosted runners
- Making it slightly more expensive to use self-hosted runners
- There is actually a fourth one: the vnet integration, which also allows you to run public runners in your own infra
As a bonus, for some people it means something that was free is now not free. Those who are willing to pay rather than go, might prefer to use GitHub-hosted if they are going to pay anyway.
This is clearly an incentive to use github-hosted, and their sales reps are also going this way.
Well, these people earn their living by saying these things that only seem to make sense superficially but don't withstand closer scrutiny.
reply▲reply▲I LOVE gitlab, but their new pricing is absurd. It feels like they are trying to shovelware their AI stuff. Their cheapest plan is more than 7x the cost of github, AND more expensive than github enterprise! And thats on the _cheapest_ non free gitlab plan.
If you self host gitlab entirely, you can't even get branch/force-push protection. If they could bring their pricing to even just 2x github by having a NON-AI plan, I would purchase again in a heartbeat.
reply▲notnullorvoid21 hours ago
[-] I had to go check to see what their pricing was, and I couldn't believe it. The base tier was $4/month, now that tier is gone and the premium tier is 2x what it used to be only 5 years ago.
reply▲GitLab CI is _excellent_. Github Actions has come a long way, but a few years back it was absolutely painful working with GA when I had GitLab CI for reference.
reply▲used to self-host gitlab CI runners around the same year also for our long running CI's due to db migrations + prepared data loading for tests.
we rent 7*4$ VPS, install gitlab CI runners on them, saving us from hundreds $$$ per month and 45mins/merge (with test running on main branch only) to 7*4$/month and 7-9mins/commit (yes, we run full test on each commit and let gitlab auto-cancel older one).
with bonus: FE team get live version of their changes on each MR.
* its 7 VPS because we separated the tests by modules and we have 7 major modules.
* edit: formatting
GitLab CI is quite good. Have been using it for several years.
reply▲I can't tolerate it.
The split between tag and branch pipelines seems like intentional obfuscation with no upsides (you can't build non-latest commit from a branch, and when you use a tag to select the commit, GitLab intentionally hides all branch-related info, and skips jobs that depend on branch names).
"CI components" are not really components, but copy-paste of YAML into global state. Merging of jobs merges objects but not arrays, making composition unreliable or impossible.
The `steps` are still unstable/experimental. Composing multiple steps either is a mess of appending lines of bash, or you have go all the way in the other direction and build layered Docker images.
I could go on all day. Programming in YAML is annoying, and GitLab is full of issues that make it even clunkier than it needs to be.
Agreed. I worked with Gitlab CI on the daily from 2021 till 2024 and I started curating a diary of bugs and surprising behavior I encountered in Gitlab.
No matter what I did, every time I touched our CI pipeline code I could be sure to run into yet another Gitlab bug.
BlackjackCF16 hours ago
[-] This is also my experience with GitLab CI.
It’s great if you have relatively simple CI. If you have anything slightly more complicated (like multiple child pipelines for a monorepo) you’re going to have a rough time.
Every time I thought I understood GitLab CI, it would fail/behave in non-obvious ways.
Yikes! They seem to be gunning for services like WarpBuild, which we've used for a couple years to keep our costs low. The $0.002 per minute on top of WarpBuild's costs is exactly GitHub's new pricing scheme.
I'm happy for competition, but this seems a bit foul since we users aren't getting anything tangible beyond the promise of improvements and investments that I don't need.
The lever that matters the most with the new $0.002/min tax is to reduce the number of minutes consumed.
Given that GitHub runners are still slow as ever, it actually is a point in our favor even compared to self-hosting on aws etc. However, it makes the value harder to communicate <shrug>.
Thanks for the email and the reminder that we can use fewer shards with larger runners.
reply▲reply▲Very cool! Are there any blog posts or articles about Tangled? Docs seem pretty light.
reply▲How does this compare to just running Hydra yourself?
reply▲Useless for Mac or Windows presumably.
reply▲What do you mean? This is for CI, whatever the dev machine runs is irrelevant. Either way, the CI uses OCI containers built with Nix, you don't need Nix installed on the host. Also Nix supports MacOS.
reply▲I wasn't talking about developers machines. If you want to build and test software on Windows or Mac in CI, you obviously can't use a CI system that requires Linux. Yes I know about cross-compilation.
> Also Nix supports MacOS.
Doesn't matter. Tangled CI requires Docker.
self-hosted, so same story?
I'm not a fan of nix and would have picked containers regardless for a git forge CI offering
It is containers. It's based on Nixery, which is a virtual Docker registry where it puts together containers for you on the fly containing whatever packages you want from Nixpkgs.
reply▲By containers I mean I get to define my CI environment with containers, not nix or that somewhere under the hood containers are being used
Like Argo or Jenkins. Pushing nix as the DX for GHA equivalent was a poor choice by Tangled IMO. It's too unusual for your average dev, I'm not interested in learning nix so I can use CI.
Seriously. They're charging me for using MY cpus?
Forgejo incoming testing period..
reply▲A lot of server software does that. People were paying absurd prices for fast Xeons to save on their Oracle bills.
reply▲PunchyHamster21 hours ago
[-] Reminds me of a customer that had in their contract requirements GHz amount so after we won the contract we digged out some old P4 based Xeons (everything after for a long time had lower clocks) and they got their stuff ran on old junk because it would be breach of contact not to.
It was govt thing and they are required to put a new bid every few years and their bid was EVIDENTLY "just list what our current hosting provider has, we can't be arsed to spend months migrating infrastructure every few years", but the clever weasels in the sales managed to get them.
It’s not unheard of, similarish to many core licensing schemes. Like mssql.
reply▲gabrielgio22 hours ago
[-] Not the same thing. The equivalent would be mssql charging by web server connections to it.
reply▲In some sense, core licensing is worse, in that you are also paying for idle capacity. But when you try to scale by activity, I think you will see it is not that much different.
reply▲Zig's decision to ditch GitHub actions seems remarkably prescient, no?
reply▲yes ! i'm actually doing the same as i saw the safe_sleep.sh code on their runners ... insane story ...
reply▲Can you elaborate on what you're referring to? Sounds interesting.
reply▲reply▲QuercusMax23 hours ago
[-] Wow, that's horrifyingly bad. Like "didn't think about it for more than 30 seconds before committing" bad.
reply▲templar_snow23 hours ago
[-] Absolutely ridiculous. Just absolutely abhorrent and downright abusive move on Microsoft's part.
reply▲> abhorrent and downright abusive move
Is it that egregious?. I read it as they are redistributing the costs. It is in combination dropping the managed runner costs by a good margin and charging for the orchestration infrastructure. The log storage and real time streaming infra isn't free for them (not $84/month/runner expensive perhaps but certainly not cheap )
We don't need to use the orchestration layer at all, even if we want to use rest of the platform, either for orchestration or runners. Github APIs have robust hooks(not charged extra) and third-party services(and self-hostable projects) already provide runners, they will all add the orchestration layer now after this news.
--
Competition is good, free[2] kills competition. Microsoft is the master of doing that with Internet Explorer or Teams today.
Nobody was looking at doing the orchestration layer because Github Actions was good enough at free[1], now the likes of BuildJet, Namespace Labs etc are going to be.
[1] Scheduler issues in Github Actions not withstanding, it was hard to compete against a free product that costs money to build and run.
[2] i.e. bundled into package pricing,
templar_snow15 hours ago
[-] I can definitely hope for more competition and new providers, yes - good point
reply▲reply▲reply▲The operate on less than 100_000€/year, so I would cut them some slack ;-)
btw. codeberg is not a company, more like a foundation, Verein is the german word.
reply▲I fail to see how this is relevant to the point I was making. With uptime being so low, it's not a viable alternative: budget / resources / etc. don't change that fact.
Don't get me wrong — I am glad that they are doing what they're doing, but it's a long way until it becomes a real alternative.
array_key_first23 hours ago
[-] We don't actually know what githubs uptime is because they don't say. If you told me it's less than 96.5, I might believe you, particular for actions.
reply▲I'm sure if Codeberg had equivalent resources they'd be good, hard to fault a nonprofit for not benefiting from a trillion dollar multinational corporation. What was GitHub's excuse for their failures?
reply▲I am not criticizing Codeberg, I'm just adding context that seems very important from my experience.
As mentioned above — I am glad that they exist and do what they do. However, that doesn't mean I should ignore issues like this.
Are they a profit center yet?
reply▲Or just run a self hosted GitLab open source instance? Running it for 6 years now with nearly zero issues and a great CI.
reply▲Last time I self hosted GitLab I was greeted with an "Update ASAP" Badge on the admin page basically every week and that has scared me away.
reply▲I was born in 1993. I kind of heard lots of rumbling about Microsoft being evil as I grew up, but I wasn't fully understanding of the anti trust thing.
It used to suprise me that people saw cool tech from Microsoft (like VSCode) and complain about it.
I now see the first innings of a very silly game Microsoft are going to start playing over the next few years. Sure, they are going to make lots of money, but a whole generation of developers are learning to avoid them.
Thanks for trying to warn us old heads!
Microsoft had a very fair shot at redeeming themselves, but with how Teams, GitHub and all the AI crap they push into GitHub and Windows, it's clear they have not changed one bit.
reply▲They did change a lot. Previously Microsoft actually cared about its main product lines. They did lots of anticompetitive things to get people onboarded. Being anticompetitive and making products that deeply bundled stuff was their evil badge not hypetrain rugpulls. However, they were adding features developers and sysadmins wanted. That's how so many businesses got Active Directory. There is still no equivalent alternative to AD. There are subsets but no equivalent set of the complete featureset. After Ballmer the company changed.
Microsoft of Nadella is different. It looks more like a boring Silicon Valley monopoly. They had good products years ago and it got people hooked and now its a game of endless rugpulls. Microsoft of now doesn't care about the featureset. They just jump from one hype train to another. People keep paying them for the stuff they did in early 2000s. Nobody cares about newer stuff including Microsoft themselves.
I’m genuinely excited about this. The GitHub actions platform is genuinely bad compared to circle or Travis but they’ve been totally crowded out because GitHub was just so easy to use. This has led to plenty of security issues and a general lack of innovation in the ci space. Hopefully by this pricing structure change we’ll see more investment in ci tooling across the industry
reply▲GitHub Actions were never too easy to use. But they were
cheap to use, so anything more expensive had trouble competing.
Now the playing field is more level, yay. Fun for those who choose to migrate away.
This. There are plenty of good/better CICD solutions out there, but it's tricky to compete with "comes for free with our VCS". I guess it's clear now there is no such thing as a free lunch. I feel it's a good thing for the "CICD industry" that people will be looking around to alternatives, and do a honest Total Cost of Ownership analysis.
reply▲Github actions are expensive. However, they were free for public repos and you could self-host. That’s what made them “cheap”.
reply▲So, let me get this straight, the "platform fee" is baked into the runner cost, but, their cheapest runner is the _same price_ as the platform fee? So its the same price to have them run it vs have me run it?
reply▲It seems like a solid plan to me:
- charge the same you would pay for the GitHub runners
- you have to factor YOUR server cost also, so self hosted will cost more than the platform option
- you jump to the platform runners and save on servers, sysadmin, DevOps, etc.
And then they grab you by the balls and raise the prices.
This seems backwards. Why charge for me to run the thing myself instead of them?
reply▲GitHub has still been managing the orchestration and monitoring of runs that you run on your own (or other cloud) hardware. They have just decided that they are no longer going to do this for free.
So the question becomes: is $0.002/minute a good price for this. I have never run GitHub Actions, so I am going to assume that experience on other, similar, systems applies.
So if your job takes an hour to build and run though all tests (a bit on the long side, but I have some tests that run for days), then you are going to pay GitHub $.12 for that run. You are probably going to pay significantly more for the compute for running that (especially if you are running on multiple testers simultaneously). So this does not seem to be too bad.
This is probably going to push a lot of people to invest more in parallelizing their workloads, and/or putting them on faster machines in order to reduce the number of minutes they are billed for.
I should note that if you are doing something similar in AWS using SMS (Systems Management Service), that I found that if you are running small jobs on lots of system that the AWS charges can add up very quickly. I had to abandon a monitoring system idea I had for our fleet (~800 systems) because the per-hit cost of just a monitoring ping was $1.84 (I needed a small mount of data from an on-worker process). Running that every 10 minutes was going to be more than $250/day. Writing/running my own monitoring system was much cheaper.
As a solo Founder who recently invested in self-hosted build infrastructure because my company runs ~70,000 minutes/month, this change is going to add an extra $140/month for hardware
I own. And that's just today; this number will only go up over time.
I am not open to GitHub extracting usage-based rent for me using my own hardware.
This is the first time in my 15+ years of using GitHub that I'm seriously evaluating alternative products to move my company to.
But it is not for hardware you own. It is for the use of GutHubs coordinators, which they have been donating the use of to you for free. They have now decided that that service is something they are going to charge for. Your objection to GitHub "extracting usage-based rent from me" seems to ignore that you have been getting usage of their hardware for free up to now.
So, like I said, the question for you is whether that $140/month of service is worth that money to you, or can you find a better priced alternative, or build something that costs less yourself.
My guess is that once you think about this some more you will decide it is worth it, and probably spend some time trying to drive down your minutes/month a bit. But at $140 a month, how much time is that worth investing?
No. It is not worth a time-scaled cost each month for them to start a job on my machines and store a few megabytes of log files.
I'd happily pay a fixed monthly fee for this service, as I already do for GitHub.
The problem here is that this is like a grocery store charging me money for every bag I bring to bag my own groceries.
> But at $140 a month, how much time is that worth investing?
It's not $140/month. It's $140/month today, when my company is still relatively small and it's just me. This cost will scale as my company scales, in a way that is completely bonkers.
> The problem here is that this is like a grocery store charging me money for every bag I bring to bag my own groceries.
Maybe they can market it as the Github Actions corkage fee
__turbobrew__22 hours ago
[-] > It is not worth a time-scaled cost each month for them to start a job on my machines and store a few megabytes of log files
If it is so easy why don’t you write your own orchestrator to run jobs on the hardware you own?
> The problem here is that this is like a grocery store charging me money for every bag I bring to bag my own groceries.
This is an odd take because you're completely discounting the value of the orchestration. In your grocery store analogy, who's the orchestrator? It isn't you.
featherless22 hours ago
[-] Do you feel that orchestration runs on a per-minute basis?
reply▲As long as they're reserving resources for your job during the period of execution, it does.
reply▲featherless22 hours ago
[-] Charging people to maintain a row in a database by the minute is top-tier, I agree.
reply▲If you really think that's all it is, I would encourage you to write your own.
reply▲featherless21 hours ago
[-] It would be silly to write a new one today. Plenty of open source + indy options to invest into instead.
For scheduled work, cron + a log sink is fine, and for pull request CI there's plenty of alternatives that don't charge by the minute to use your own hardware. The irony here, unfortunately, is that the latter requires I move entirely off of GitHub now.
PunchyHamster21 hours ago
[-] so they are selling cent of their CPU time for a minute's worth
> My guess is that once you think about this some more you will decide it is worth it, and probably spend some time trying to drive down your minutes/month a bit. But at $140 a month, how much time is that worth investing?
It's $140 right now. And if they want to squeeze you for cents worth of CPU time (because for artifact storage you're already paying separately), they *will* squeeze harder.
And more importantly *RIGHT NOW* it costs more per minute than running decent sized runner!
I get the frustration. And I’m no GitHub apologist either. But you’re not being charged for hardware you own. You’re being charged for the services surrounding it (the action runner/executor binary you didn’t build, the orchestrator configurable in their DSL you write, the artefact and log retention you’re getting, the plug-n-play with your repo, etc). Whether or not you think that is a fair price is beside the point.
That value to you is apparently less than $140/mo. Find the number you’re comfortable with and then move away from GH Actions if it’s less than $140.
More than 10 years of running my own CI infra with Jenkins on top.
In 2023 I gave up Jenkins and paid for BuildKite. It’s still my hardware. BuildKite just provides the “services” I described earlier. Yet I paid them a lot of money to provide their services for me on my own hardware. GH actions, even while free, was never an option for me. I don’t like how it feels.
This is probably bad for GitHub but framing it as “charging me for my hardware” misses the point entirely.
feels like a new generation is learning what life is like when microsoft has a lot of power. (tl;dr: they try to use it.)
reply▲I was born in 1993. I kind of heard lots of rumbling about Microsoft being evil as I grew up, but I wasn't fully understanding of the anti trust thing.
It used to suprise me that people saw cool tech from Microsoft (like VSCode) and complain about it.
I now see the first innings of a very silly game Microsoft are going to start playing over the next few years. Sure, they are going to make lots of money, but a whole generation of developers are learning to avoid them.
Thanks for trying to warn us old heads!
PunchyHamster21 hours ago
[-] Feels like listening to Halo generation being surprised MS fucks them over, because they thought they were Good Guys, coz they Made Thing They like
reply▲Yeah, I'm no GitHub apologist, but I'll be one in this context. This is actually a not-unreasonable thing to charge for. And a price point that's not-unreasonable.
It makes sense to do usage-based pricing with a generously-sized free tier, which seems to be what they're doing? Offering the entire service for free at any scale would imply that you're "paying" for/subsidizing this orchestration elsewhere in your transactions with GitHub. This is more-transparent pricing.
Although, this puts downward pressure on orgs' willingness to pay such a large price for GH enterprise licenses, as this service was hitherto "implicitly" baked into that fee. I don't think the license fees are going to go down any time soon, though :P
I run about 1 action a day taking 18h running on 2 runners
One being self hosted 24gb ram 8 core ARM vps and one being a 64gb 13900k x86 dedicated server
Now the GitHub pricing change definitely? costs more than both servers combined a month ... (They cost about 60$ together )
3 step GitHub action builds around 1200 nix packages and derivations , but produces only around 50 lines of logs total if successful and maybe 200 lines of log once when a failure occurs
And I'm supposed to pay 4$ a day for that ?
Wonder what kind of actual costs are involved on their side of waiting for a runner to complete and storing 50 lines of log
alexellisuk13 hours ago
[-] It sounds like you'd be better off self-hosting Jenkins. The other issue with GHA is they cap all runs at 6 hours.
Despite what people say about "maintaining" Jenkins (whatever that means to them personally) - you can set it up in an IaaC way including the jobs. You can migrate/create jobs en masse via its API (I did this about 10 years ago for a large US company converting from what was then called TFS)
I'll likely check out buildbot or just switch to gitlab
reply▲franktankbank4 hours ago
[-] What problem does Jenkins solve? When we got jenkins working how we wanted it was a giant groovy script that was handling checkout manually.
reply▲Somewhere around 0.00004$ probably.
Nice profit margin…
You know, one might ask what the base fee of $4k/mo (in my org's case) is covering, if not the control plane?
Unless you're on the free org plan, they're hardly doing it "for free" today…
Exactly this. It’s not like they don’t have plenty of other fees and charges. What’s next, charging mil rates for webhook deliveries?
reply▲dragonwriter22 hours ago
[-] > They have just decided that they are no longer going to do this for free.
Right, instead, they now charge the full cost of orchestration plus runner for just the orchestration part, making the basic runner free.
(Considering that compute for "self-hosted" runners is often also rented from some party that isn't Microsoft, this is arguably leveraging the market power in CI orchestration that is itself derived from their market power in code hosting to create/extend market power in compute for runners, which sounds like a potential violation of both the Sherman Act and the Clayton Act.)
Sure, but that shouldn't be a time-dependent charge. If my build takes an hour to build on GH's hardware, sure thing, charge me for that time. But if my build takes an hour to build on _my_ hardware, then why am I paying GH for that hour?
I get being charged per-run, to recoup the infra cost, but what about my total runtime on my machine impacts what GH needs to spend to trigger my build?
> is $0.002/minute a good price for this
Absolutely not, since it's the same price as their cheapest hosted option. If all they're doing is orchestration, why the hell are they charging per-minute instead of per-action or some other measure that recognizes the difference in their cost between self-hosted and github-hosted?
> is $0.002/minute a good price for this
It was free, so anything other than free isn't really a good price.
It's hard to estimate the cost on github's side when the hardware is mine and therefore accept this easily.
(Github is already polling my agent to know it's status so whether is "idle" or "running action" shouldn't really change a lot on their side.)
...And we already pay montly subscription for team members and copilot.
I have a self-hosted runner because I must have many tools installed for my builds and find it kinda counter productive to always reinstall those tools for each build as this takes a long time.
(Yeah, I know "reproducible builds" aso, but I only have 24h in most of my days)
Even for a few hundreds minutes a month, we're still under a few $ so not worth spending two days to improve anything... yet.
Is it polling the runner, or is the runner sending it progress?
reply▲ExoticPearTree22 hours ago
[-] The runner sends progress info, polls for jobs and so on. The runners don't have to be accessible from GitHub, they just needs general internet access (like through a NAT device).
reply▲> GitHub has still been managing the orchestration and monitoring of runs that you run on your own (or other cloud) hardware. They have just decided that they are no longer going to do this for free.
This argument is disingenuous. Companies pay GitHub per seat for access to PR functionality etc. What's next, charging per repository? Because of a decision to no longer provide the repositories "for free"? It's not for free, you're paying already, it's included in the per-seat pricing. If you charge per seat then sometimes there are users who hardly use it and sometimes there are users who use it a lot. The per-seat pricing model is supposed to make the service profitable overall regardless of the usage levels of individual users.
> $0.002/minute a good price for this.
It is not only not good. It is outrageous. The amount of compute required for orchestration is small (async operations) and also they already charge your for artifacts storage. You need to understand that the orchestration just receives details (inbound) from the runner. It needs very little resources.
Additionally, they could just self-host their code since code is data is a moat.
reply▲Because they know Forgejo is starting to get attention from major players and thus becoming competitive, and hosting your own CI infrastructure will make completely moving away from GitHub all that easier - If you don't really care about the metadata all it pretty much takes is moving git repositories with their history.
Or shortly summarized: lock in through pricing.
Pretty sure this will explode straight in their faces though. And pretty damn hard.
How can you lock in through charging money?
Seems it’s like the opposite and they are charging because people are already locked in and they can or am I misreading your comment?
reply▲Microsoft "suddenly" does not seem to want you to run your own CI, which is a key part of running your own SCM. And this decision miraculously happens the moment a lot of big orgs are looking at self-hosting a cost effective (because open source) near 1:1 alternative to GitHub (=Forgejo).
So they make CI a bit cheaper but a future migration to Forgejo harder.
In fact they could easily pull off some typical sleazy Microsoft bullshit and eventually make it a shit ton harder to migrate out of GitHub once you migrated back in.
The idea is that they let you stay locked in for free. They dissuade people from making their CI pipeline forge-agnostic by charging you if you if you take steps to not be dependent on them. This means they can keep charging in other areas, and keep people in GitHub so that it stays dominant. Dominance is something that can be used to keep people in the Microsoft ecosystem, keep GitHub as the place where code goes so they have training data for LLMs, and dominance can simply be cashed in down the line.
I don’t know if that’s actually why they’re doing this, but it sounds plausible.
dragonwriter22 hours ago
[-] If you make running your own runners as expensive as running on Github's runners
on top of the cost of actually hosting the runners, then if you are currently on Github and not able to migrate off immediately, the price conscious decision is to migrate runners
into Github. But then, its even harder if you ever decide to migrate your whole operation out.
Now, if you are already looking at migrating, its also potentially a kick in the butt to do it now. But if you aren’t, the path of least resistance—or at least, the path of least present recurring cost—is a path to a greater degree of lock-in.
I don't think Forgejo is competitive in the markets GitHub makes most of their money from, nor does it seem Forgejo developers want it to be.
reply▲parliament3223 hours ago
[-] Where does GitHub even make most of their money? Their compliance posture makes them a non-starter for any regulated industries (which is atypical for a Microsoft property, generally MS is the market leader for compliance in all of their products).
reply▲Places might be officially regulated, but neither government agencies, healthcare, finance or defense industries are as strict as you think. People have to get stuff done, and most are usually quite incompetent in these protected industries.
Microsoft’s sales reps know this.
Given that a lot of places that deal with money use them, I find your comment quite interesting and would like to learn more :)
reply▲Representatives from the Dutch government recently had a chat with representatives from Forgejo because they are quite interested in migrating their SCM infrastructure from Github to Forgejo.
And trust me, they are running a lot of public and private repositories.
And there are many more orgs and govs throughout Europe doing similar things because there's a (growing) zeitgeist here that the Trump administration nor any American SaaS company can be trusted. This started, by the way, after Microsoft suspended the ICJ from using Microsoft 365 on orders from the White House.
Can confirm.
I have seen this sentiment more and more, which is welcome to me as it’s a drum I have been banging for 15 years.
I have never had so many empathetic conversations than I have recently.
Sounds familiar!
Everybody now is like "Hey, we can take something like Kubernetes which is open source and is backed by a worldwide community, and you know like OpenStack which is open source and is backed by a worldwide community and we can build our own computing platform and deploy services and online communities and stuff on top of that"
And I was like "Wait, you guys are realizing that NOW?!? I've been an activist and part of a movement urging you all to try and be less dependent on US Big Tech and focus more on decentralization for YEARS"
Like you I am really happy things seem to get rolling now, though :)
The Dutch government represenrative mentioned contacts with French colleagues about this also.
reply▲PunchyHamster21 hours ago
[-] Not sure why you think forgejo is competition and not Gitlab.
> Or shortly summarized: lock in through pricing.
how would increasing price make you locked in more ?
> If you don't really care about the metadata all it pretty much takes is moving git repositories with their history.
moving PR/CI/CD/Ticket flow is very significant effort, as in most companies that stuff is referenced everywhere. Having your commits refer ticket ID from system that no longer exists is royal PITA
> Having your commits refer ticket ID from system that no longer exists is royal PITA
just rewrite the short links in your front-end to point to the migrated issues/PRs. write a redirect rule for each migrated issue/PR, easy
hard-coded links in commit messages are annoying, you can redirect in the front-end too but locally you'd have to smudge/clean them on local checkout/commit
I would keep repos on GH but use Jenkins though.
reply▲Because GHA was stagnant and expensive and multiple services like
https://www.warpbuild.com/ popped up, with better performance and much lower price. Looks like they ate enough of GH’s lunch…
Hey, WarpBuild founder here.
While it makes it harder for us to communicate this, we're still, we're still faster and cheaper even after the $0.002/min self hosting tax.
Overall costs go up for everyone but we remain the better option.
They still run the whole orchestration.
If you don't want to pay, you'd have to not use GitHub Actions at all, maybe by using their API to test new commits and PRs and mark them as failed or passed.
One problem is that GitHub Actions isn't good. It's not like you're happily paying for some top tier "orchestration". It's there and integrated, which does make it convenient, but any price on this piece of garbage makes switching/self-hosting something to seriously consider.
reply▲Github being a single pane of glass for developers with a single login is pretty powerful. Github hosting the runners is also pretty useful, ask anyone who has had to actually manage/scale them what their opinion is about Jenkins is. Being a "Jenkins Farmer" is a thankless job that means a lot of on-call work to fix the build system in the middle of the night at 2am on a Sunday. Paying a small monthly fee is absolutely worth it to rescue the morale of your infra/platform/devops/sre team.
Nothing kills morale faster than wrenching on the unreliable piece of infrastructure everyone hates. Every time I see an alert in slack github is having issues with actions (again) all I think is, "I'm glad that isn't me" and go about my day
I run Jenkins (have done so at multiple jobs) and it's totally fine. Jenkins, like other super customizable systems, is as reliable or crappy as you make it. It's decent out of the box, but if you load it down with a billion plugins and whatnot then yeah it's going to be a nightmare to maintain. It all comes down to whether you've done a good job setting it up, IMO.
reply▲Lots of systems are "fine" until they aren't. As you pointed out, Jenkins being super-customizable means it isn't strongly opinionated, and there is plenty of opportunity for a well-meaning developer to add several foot-guns, doing some simple point and click in the GUI. Or the worst case scenario: cleaning up someone elses' Jenkins mess after they leave the company.
Contrast with a declarative system like github actions: "I would like an immutable environment like this, and then perform X actions and send the logs/report back to the centralized single pane of glass in github". Google's "cloud run" product is pretty good in this regard as well. Sure, developers can add foot guns to your GHA/Cloud Run workflow, but since it is inherently git-tracked, you can simply revert those atomically.
I used Jenkins for 5-7 years across several jobs and I don't miss it at all.
Yeah, it seems like a half-assed version of what Jenkins and other tools have been doing for ages. Not that Jenkins is some magical wonderful tool, but I still haven't found a reasonable way to test my actions outside of running them on real Github.
reply▲Everyone who has Actions built into their workflow now has to go change it. Microsoft just conned a bunch more people with the same classic tech lock-in strategy they've always pursued, people are right to be pissed. The only learning to take away is never ever use anything from the big tech companies, even if it seems easier or cheaper right now to do so, because they're just waiting for the right moment to try and claw it back from you.
reply▲> Microsoft just conned a bunch more people with the same classic tech lock-in strategy they've always pursued, people are right to be pissed
People would be better served by not expecting anything different from Microsoft. As you say yourself, this is how they roll.
> The only learning to take away is never ever use anything from the big tech companies
Do you even believe in this yourself? Not being dependent on them would be a good start.
Yep, this mostly works fine (and can be necessary already in some setups anyway), the main issues are that each status update requires an API call (over v3, AFAIK updating statuses was never added to v4) so if you have a lot of statuses and PR traffic you can hit rate limits annoyingly quickly, and github will regularly fail to deliver or forward webhooks (also no ordering guarantees).
reply▲nextaccountic10 hours ago
[-] I mean, is there some open source project that already uses webhooks to replace Github Actions?
Rather than having to write some ad hoc code to do this
We have internal integrations with GitHub webhooks that will hit our server to checkout a branch, run some compute, and then post a comment on the thread. Not sure if you can integrate something like that to help block a PR from being merged like Actions CI checks, but you can receive webhooks and make API calls for free (for now). Would definitely result in some extra overhead to implement outside of Actions for some tasks.
reply▲> Not sure if you can integrate something like that to help block a PR from being merged like Actions CI checks
Post statuses, and add rulesets to require those statuses before a PR can be merged. The step after that is to lock out pushing to the branch entirely and perform the integration externally but that has its own challenges.
Because they make money from charging way over cost price for per-minute CI runners, and they don't want people using much much cheaper alternative providers.
They don't care about people actually self-hosting. They care about people "self hosting" with these guys:
https://github.com/neysofu/awesome-github-actions-runners
Because charging you brings more profits than not charging you.
reply▲Because they host the artifacts, logs, and schedule jobs which run on your runners, I assume.
reply▲Then why do they charge by the minute instead of gigabytes and number of events?
reply▲Ask them. I don’t set the policy at a company I don’t work at.
Their announcement gives a clue, and it’s to do with job orchestration.
they charge you for artifacts and logs separately, already
reply▲Yep and the sky is blue and GitHub can charge for that too if they want to.
I don’t make policy at GitHub and I don’t work at GitHub so go ask GitHub why they charge for infrastructure costs like any other cloud service. It has to do with the queueing and assignment of jobs which is not free. Why do they charge per minute? I have no idea, maybe it was easiest to do that given the billing infrastructure they already have. Maybe they tried a million different ways and this was the most reasonable. Maybe it’s Microsoft and they’re giving us all the middle finger, who knows.
I don't think you're responsible for anything more than your own comments.
I added some context that contradicts your assumption that the increased fees were to cover hosting/storage/scheduling costs.
The scheduler isn’t free, I always wondered how the financials work on this one. Turns out they didn’t ;)
Anyway, GitHub actions is a dumpster fire even without this change.
I develop software, I also test and run it. All in my machines.
But you (yes, you personally) have to collect the results and publish them to a webpage for me. For free.
Would you make this deal?
It sounds like a bad deal right?
Except the alternative is I do this for free but also I'm doing all the testing and providing the hardware.
I'm only going to charge you if you do most of the work yourself
If you do it all, you can optimize the whole supply chain. Maybe you can put some expensive capacity you built to use and leverage it when otherwise impossible, etc.
Maybe it's bad business dealing with lots of non-standardized external hosts, and it drags you down.
Maybe people are abusing the free orchestration to do non-CI stuff and they're compromising legitimate users.
Look, I understand it's frustrating to some consumers. However, it's not irrational from GitHub's point of view.
This is actually about abusing Microsoft's market position to eliminate competitors in related markets, plain & simple.
reply▲if you were paying me a monthly license fee for each developer working on your repos, I'd probably consider it
reply▲What happens if I am, and now my developers suddenly start to produce changes much faster? Like, one developer now produces the volume of five.
Would you keep charging the same rate per head?
justcool39314 hours ago
[-] why wouldn't you? these are easily compressible text files. storing even like 100x into a 400 day (at most, the default for GH is 90) box is downright cheap to do on even massive scales.
it's 2025, for log files and a spicy cron daemon (you pay for the artifact storage), it's practically free to do so. this isn't like the days of Western Union where paying $0.35 to send some data across the world is a good deal
If that's the case, why all the fuzz?
All the people complaining can just tap into this almost-free and acessible cheap resource you are referring to instead.
we don't need it. we need to run our CI jobs on resources we manage ourselves, and GitHub have started charging per-minute for it. apples and cannonballs
reply▲no, I'd cut the monthly seat cost and grow my user base to include more low-volume devs
but realistically, publishing a web page is practically free. you could be sending 100x as much data and I would still be laughing all the way to the bank
Publishing the page is only the last step. It's orchestrating the stuff THEN publishing it.
If you think that's easy, do it for me. I have some projects to migrate, give me the link of your service.
> If you think that's easy
I think it's cheap to maintain. let me know how many devs you have, how many runs you do, and how many tests (by suite) you have, and I can do you up a quote for hosting some Allure reports. can spread the up-front costs over the 3-year monthly commitment if it helps
There are several services I know who offer this for free for open source software, and I really doubt any commercial offerings of that software would charge you extra for what is basic API usage.
reply▲But I get to read all your code and use it for training my AI, right?
reply▲My projects are public anyway. If you respect the license and make the AI comply to valid license reuse, I'm game.
reply▲> My projects are public anyway.
My point was that they profit from accessing your code, which is why they made it free in the first place. Now they make you pay because they believe they will make more profit. But they certainly weren't losing money before.
> If you respect the license and make the AI comply to valid license reuse
I think that the de facto situation is that AI does not have to know about licences or copyright at all. If they hack your computer to train their AI, the illegal part is that they hacked your computer, not that they trained their AI with the stolen data.
> Companies can get into legal trouble if they don't.
Heard of Meta torrenting copyrighted material? What kind of trouble did they get into?
> Coming soon: Simpler pricing and a better experience for GitHub Actions
i think it should be illegal or otherwise extremely damaging to do this kind of thing
Come on, editorializing the post title is against HN guidelines, but making it illegal is a bit too harsh.
reply▲Why would the self-hosted runner fee be per-minute instead of per-job? I don’t get it.
reply▲I had the same question — I understand that the Actions control plane has costs on self-hosted runners that GitHub would like to recoup, but those costs are fixed per-job. Charging by the minute for the user’s own resources gives the impression that GitHub is actually trying to disincentivize third-party runners.
reply▲Self-hosted runner regularly communicates with the control plane, and control plane also needs to keep track of job status, logs, job summaries, etc.
8h job is definitely more expensive to them than a 1 minute one, but I'd guess that the actual reason is that this way they earn more money, and dissuade users from using a third party service instead of their own runners.
Might be an estimation of logs storage/bandwidth.
reply▲That's generous, but doesn't seem consistent with how Microsoft does business. Also, if that's the case why does self-hosted cost the same as the lowest hosted tier?
reply▲or some other metric like how many logs your job produces and they have to process
the only rational outside rationale is this has the best financial projections, equitability with the customer be damned
gotta make up for those slumping ai sales somehow, amiright?
franklyworks17 hours ago
[-] Runner price based on CPU/memory and time makes sense, since those are the costs associated with executing runners.
The costs for GitHub doing action workflows (excluding running) is less related to job duration.
The most charitable interpretation is that per-minute pricing is easier to understand, especially if you already pay runners per minute.
The less charitable interpretation is that they charge that because they can, as they have the mindshare and network effect to keep you from changing.
Here are the practical implications and considerations to optimize for cost, given the new pricing. These are generic and ensure you think through your workflows and runners before making any changes.
1. Self-hosting runners is still cheaper than not
Despite the $0.002/minute self-hosted runner tax, self-hosting runners on your cloud (aws/gcp/azure/...) remains the cheaper option.
2. Prefer larger runners
If your workflow scales with the number of vCPUs, prefer larger runners. That ensures you spend fewer minutes on the runner, which reduces the GitHub self-hosted runner tax.
For example, using actions-runner-controller with heavy jobs running on 1 vcpu runners is not a good idea. Instead, prefer a 2vcpu runner (say) if it runs the job ~2x faster.
3. Prefer faster runners
All else being equal, prefer faster runners. That ensures you spend fewer minutes on the runner, which reduces the GitHub self-hosted runner tax.
For example, if you're self-hosting on aws and using a t3g.medium runner, it's better to use a t4g.medium runner since the newer generation is faster, but not much more expensive.
4. Prefer fewer shards
If you have a lot of shards for your jobs (example: tests on ~50 shards), consider reducing the number of shards and parallelizing the tests on fewer but larger runners.
5. Improve job performance
This is not new advice, but it's now more important than ever because of the additional GitHub self-hosted runner tax.
6. Use GitHub hosted runners for very short jobs
For linters and other very short jobs, it's better to use GitHub hosted runners.
Note: I make WarpBuild, where we provide github actions runner compute. Our compute is still cheaper than using github hosted runners (even with the $0.002/min tax) and our runners are optimized for high performance to minimize the number of mins consumed.
I'm generally biased, but I think the points 1-6 apply irrespective of WarpBuild.
franklyworks17 hours ago
[-] Any thoughts on what to extent GitHub is subsidizing OSS development with its CI?
This feels like one of the big issues that OSS projects might face when migrating to an alternative.
What might a less GitHub centric CI ecosystem look like for OSS community?
Small to mid sized OSS projects benefit heavily from this. There is a size beyond which the free runner sizes become insufficient, but the assumption is that some form of monetization is figured out by that time.
For example, we have a lot of OSS projects using WarpBuild because performance and fast CI is important for productivity.
Without GitHub's free CI for public repos, the small projects and indies will get hit the hardest imo.
However, I do not know hard numbers to quantify the impact.
I'm happy to see they're investing in Actions — charging for it should help make sure it continues to work. It's a huge reason Github is so valuable: having the status checks run on every PR, automatically, is great. Even though I'm more of a fan of Buildkite when it comes to configuring the workflows, I still need something to kick them off when PRs change, etc.
Charging a per-workflow-minute platform fee makes a lot of sense and the price is negligible. They're ingesting logs from all the runners, making them available to us, etc. Helps incentivize faster workflows, too, so pretty customer-aligned. We use self-hosted runners (actually WarpBuild) so we don't benefit from the reduced default price of the Github-hosted runners, but that's a nice improvement as well for most customers. And Actions are still free for public repos.
Now if only they'd let us say "this action is required to pass _if it runs_, otherwise it's not required" as part of branch protection rules. Then we'd really be in heaven!
This pricing model will continue to incentivize them internally to not fix the hundreds of clearly documented issues that causes CI to be incredibly slow. Everything from their self-inflicted bottlenecking of file transfers to the safe_sleep bug that randomly makes a runner run forever until it times out. All of it now makes them more money
reply▲> charging for it should help make sure it continues to work
It's there a particular reason you're extending the benefit of the doubt here? This seems like the classic playbook of making something free, waiting for people to depend on it, then charging for it, all in order to maximize revenue. Where does the idea that they're really doing this in order to deliver a more valuable service come from?
Yeah. This is a reaction to providers like blacksmith or self-hosted solutions like the k8s operator being better at operating their very bad runner then them, at cheaper prices, with better performance, more storage and warm caches. The price cut is good, the anticompetitive bit where they charge you to use computers they don't provide isn't. My guess is that either we're all gonna move to act or that one of the SaaS startups sue.
reply▲peterldowns23 hours ago
[-] I appreciate being able to pay for a service I rely on. Using self-hosted runners, I previously paid nothing for Github Actions — now I do pay something for it. The price is extremely cheap and seems reasonable considering the benefits I receive. They've shown continued interest in investing in the product, and have a variety of things on their public roadmap that I'm looking forward to (including parallel steps) —
https://github.com/orgs/github/projects/4247?pane=issue&item....
Charging "more than nothing" is certainly not what I would call maximizing revenue, and even it they were maximizing revenue I would still make the same decision to purchase or abandon based on its value to me. Have you interacted with the economy before?
> The price is extremely cheap
and you expect it to stay this way?
peterldowns23 hours ago
[-] > and seems reasonable considering the benefits I receive.
> I would still make the same decision to purchase or abandon based on its value to me.
I don't think it makes sense to charge per minute just for logs. If they want to charge for log retention, sure, go ahead. But that is pennies, let's be real.
reply▲reply▲sadly they, afaik, do not implement the permission model. So no way to control the token permissions.
(plz correct me if i'm wrong)
Companies like Ubicloud gives hosted actions faster and far more cheaper (5-10x) than Microsoft itself.
Now Microsoft will charge "data plane usage" (CRUDing a row that contains (id, ts, state_enum, acc_id ...) in essence) 2.5 more than what Ubicloud offers for WHOLE compute. Also to have "fair pricing" they'll make you pay 2.5 more the compute's price for being able to use their data plane.
cool.
it's rather egregious that it is a "per minute" tax rather than a $0.002 per job.
reply▲I wonder how much they made from engineering practices such as
https://github.com/actions/runner/issues/3792.
To spell it out: jobs can hang forever because of some ridiculously bad code on their end, they have a 6 hour cap, so that's 6 hours of billable $$$ per-instance of the bug (assuming it wasn't manually canceled). I know I've seen jobs hang forever regularly over the course of my years using GitHub for work.
Note: pretty sure this has been resolved.
Oh, I had that happen fairly recently.
reply▲The $0.002 per-minute for self-hosted runners will definitely change the unit economics for a lot of 3rd party runner providers.
I'm sure we'll feel it too at https://sprinters.sh, but probably a bit less than others as our flat $0.01 per job fee for runners on your own AWS account will still work out to about 80% average savings in practice, compared to ~90% now when using spot instances.
I use GitHub Actions for only one thing, which is to automatically assign any issues to myself (by using the "gh" program), and I am not paying anything for it. Furthermore, the repositories that use this are all public (I do not have any private repositories on GitHub, and due to various things I will not do that).
As far as I can tell from that article, these changes will not affect me; it says "Standard GitHub-hosted or self-hosted runner usage on public repositories will remain free" and another section says "This will not impact Actions usage in public repositories". Hopefully, this information would behelpful for other people who use GitHub Actions. However, I don't know if I missed something else that is important, from the article.
This sounds correct, there are no changes for public repositories.
For private repositories, each GitHub account gets 2000 free minutes of runtime per month. Both self-hosted runners and GitHub-hosted runners count against that quota.
Microsoft has started raising prices on many of their products. I suppose they decided that their current customers need to pay the increased CapEx for AI;) New motto - AI pay for it whether you use it or not.
reply▲No such thing as free parking
reply▲No but there is validated parking for customers of other services.
This is going to be the downfall of GA
Earlier this year I priced out AWS's on-demand m7i.large instances at $0.002/minute [1]. GitHub's two-core costs $0.008/minute today so it was a nice savings. But it looks like this announcement doubles the self-hosted cost and reduces their two-core system pricing to $0.006/min.
From this perspective this is a huge price jump, but self-hosting to save money can still work out.
Honestly, GitHub Actions have been too flaky for me and I'm begrudgingly reaching for Jenkins again for new projects.
[1] https://instances.vantage.sh/aws/ec2/m7i.large?currency=USD&...
Have you considered other options like woodpecker for example?
reply▲davidpaulyoung22 hours ago
[-] Why not just self-host Gitea? CI/CD, runners, all included. Freedom. Don't have the time do keep it going and safe? No worries, folks like
https://federated.computer do that.
Forgejo might be a better option for that now.
reply▲a per-job cost instead of per-minute cost for non-compute "control plane" for CI would have made more sense and seemed more reasonable to me -- but don't really know if customers would have liked it better/worse or paid more/less under it.
(I work exclusively on public repo open source at the moment, and get Github actions for free).
Could this change mainly be about competition with their own hosted runners?
Today it's possible to spin up a company that sells GitHub Actions runners with a lower price and higher performance than GitHub's own hosted runners. These new fees will make that a lot less economically viable.
With these changes, three things hold:
1. Services like WarpBuild (I'm the founder) are still cheaper than GitHub hosted runners, even after including the $0.002/min self-hosting tax.
2. The biggest lever for controlling costs now is reducing the number of minutes used in CI. Given how slow Github's runners are, or even the ones on AWS compared to our baremetal processor single core performance + nvme disks, it makes even more sense to use WarpBuild. This actually makes a better case for moving from slow AWS instances running with actions-runner-controller etc. to WarpBuild!
3. Messaging this to most users is harder since the first reaction is that Github options make more sense. After some rational thought, it is the opposite.
Our current GitHub bill is $90/month, this would add an additional $700/month. I don't see how this doesn't cause a mass outflux.
reply▲How much do you pay for the servers that run actions? Is it much more than $610? Then it kinda makes sense.
reply▲No it's much less. Our runners are hosted in AWS ECS, you'd be surprised how affordable you can make that given the right optimizations (which is probably why GH made this decision).
reply▲Curious: Can you expand a little bit on your usage? $700/month equates to 350,000 minutes. Are you just running a truck-load of different Actions, or are the Actions themselves long-lived (waiting on something to complete)?
reply▲Just a lot of different actions, none of them long-lived. It's CI/CD for a large monorepo maintained by a relatively small team.
We use feature branch deployments, so we trigger a lot of builds.
I guess it was only a matter of time...
Part of this is fair since there is a cost to operating the control plane.
One way around this is to go back to using check runs. I imagine a third party could handle webhooks, parse the .github/workflows/example.yml, then execute the action via https://github.com/nektos/act (or similar), then post the result.
inb4 "our webhooks are now 2c per call"
reply▲I didn't find a single example of any of the upcoming features, should I follow them in github to read release notes?
reply▲they are only building "ai" features now
reply▲No, the AI is only building AI features now, or so they say.
reply▲I guess I’ll start to look at an alternative to GitHub self hosted runners.
It’s been awhile since I looked. What’s a good alternative?
Are there any good CI systems to begin with? joking, but not really
Jenkins has been rock solid, we are trying to migrate to Argo Workflows/Events, but there are a complaints (like deploying argo workflows with helm, such fun!)
regularmother23 hours ago
[-] I've been using dagger.io and it's been really nice to work with.
- runs locally
- has a language server: python, typescript, go, java, OR elixer
- has static typing
- the new caching mechanisms introduced in 0.19.4 are chef's kiss
I do not work for dagger and pay for it using the company credit card. A breath of fresh air after the unceasing misery and pain that is Gitlab and GHA.
I use Dagger as well, since v0.1.2, even worked on the CUE stuff around then with them.
I wouldn't call it a CI system though, but certainly the philosophy that local and CU should be running the same thing saves many hours of frustration.
I'm currently using Dagger to create forkable/rewindable agent sessions and environments (not with their agent nonsense). Dagger is a pretty sweet piece of tech, so many uses for programmatic container layers
"Jenkins is the worst form of CI, except for all those other forms
that have been tried."
-- Winston Churchill (probably)
buildkite
reply▲I am just waiting for GitHub starting to charge for API usage ...
reply▲On the heavy side, but TeamCity is full of goodies.
reply▲Pay even more to bring your own hardware? Well, that's new.
I get that self-hosted runners generate huge egress traffic, but this is still wild. Hope it pushes more companies to look into self-hosted Gitea / Forgejo / etc.
Jenkins not looking so bad anymore..
reply▲The email I received from them this morning claims that this will be cheaper for 96% of users...
I have cron jobs on several github projects that runs once a day and I have never been charged anything for it (other than my github membership). Should I expect to be charged for this?
I would think that majority of users does not use GitHub actions at all or have very light infrequent usage so that would be true. I think with my personal project I have never exceeded the resources they give me as part of personal pro subscription.
reply▲There has to be a VC in this thread, go ahead and fund a GitHub competitor that offers a flat monthly(yearly?) rate.
Focus on the enterprise. Something like a 3000$ minimum yearly price. Direct customer support with real engineers no questions asked.
Need someone to setup your CICD, that's another fee, but on staff engineers will get it done.
Edit: I'd even imagine a company like this can bootstrap, I'd need help though. Would probably take 4 skilled SWEs about 6 months for an MVP.
It's Jujutsu based, but I imagine East River Source Control
https://ersc.io/ may be building a GitHub competitor.
This seems totally unreasonable. How can they justify charging you based on usage when it's running on and using your resources?
reply▲sentrysapper23 hours ago
[-] Postman pulled this same stunt in 2022, limiting how many times you can run your own API class from your machine. To this day I've never reconciled with them or their product management decisions.
reply▲I wonder if players like Depot could sidestep GHA by using webhooks instead of acting as a custom runner, in other words build their own compatible control plane. I guess it would probably break a lot of workflows.
What I'd really like to see is some new CI/CD systems though. Actions is garbage in multiple dimensions. Can't somebody do something clever and save us from this flaky insecure YAML hell?
Founder of Depot[0] here. To answer your idea, at Depot we already have this concept internally. In fact, Depot isn't reliant on webhooks at all to run your jobs. One of the reasons we can be up running your jobs when GitHub webhooks service is down. Effectively, we listen to a different system to know you have a job that needs to be run.
To your second statement, I generally agree. Sounds strange to say given we're in the business of GHA runners. But it's just not a performant or reliable system at scale. This change from GitHub doesn't smell of a company that wants to do right by it's users.
If you are interested in what is up next for us at Depot, feel free to ping me via the email in my bio. I think you'll be quite interested in what we are doing.
[0] https://depot.dev
That makes me genuinely curious about the internal hosted vs. self-hosted usage ratio they're seeing. I'd have guessed the bulk of the cost/volume was on hosted, but clearly that can't be the case
reply▲Anecdotally I've seen it get a lot more common to use third-party managed runners (e.g. Blacksmith) for anyone that needs slightly beefier machines and/or a caching system that actually works.
reply▲Yeah, we migrated to self hosted actions runnrers on hetzner 2 years ago, the speed improvement was massive
reply▲GitHub charges too much for hosted runners. It's pretty straightforward to switch to another runner provider at literally half the cost of GitHub.
reply▲They are not just more expensive, they are also slower. Last time I compared them, AWS ARM64 instances could easily run jobs 30% faster, for the same CPU/memory count, than those that GitHub offers.
reply▲The way Github, Xamarin and other acquisitions have gone down, it is quite clear that the Satya charming phase is sadly gone.
reply▲We're microsoft. We don't care. We don't have to care, we're microsoft. Lock in? Embrace, expand, extinguish? Anti-competive? Anti-trust? We don't care. We don't have to care. Pay taxes? We don't have to pay taxes (
https://www.propublica.org/article/irs-microsoft-audit-back-...). We're ... etc.
This is not new, not unexpected. This is ongoing. Nothing stops this because who wins elections? How do they pay for all that publicity. Certainly "contributing" to campaigns is much cheaper than paying your taxes.
Supposedly this is a place for hackers. Hackers can build a better alternative.
Those of us who have been writing it "Micro$oft" since the 90's can now say, "I told you so."
reply▲Given they've been essentially subsidizing self-hosted orgs for a while, I'm kinda surprised they didn't do this before now. Probably wanted to lead with the price cut for everyone else.
It'll be interesting to see how this affects third party companies providing GitHub runners.
A few years ago, I had a build that was a bit slow on Github actions. I didn't want to switch to the paid plan just to spin up a worker. Basically we are a bootstrapped company with, at the time, no budget to pay ourselves or for extra stuff like fancy build servers. If you are that kind of company, Github is amazing value.
To solve the problem, I created a simple vm in Google Cloud with a lot of CPU and memory that runs Ubuntu. I installed enough stuff on it to be able to check out code and run our build script (a jvm and gradle basically). And then I modified the Github action to 1) start the vm, 2) trigger the build script via ssh 3) pause the vm so we don't get billed for it. That vm runs for maybe an hour per month or so. It would probably cost us hundreds of euros per month if we ran it 24/7. But 1/3600th of that barely registers on our bills. And it's nice and fast.
This has been working flawlessly for a few years now. The Github action takes about 3 minutes. That includes starting the vm, running the script, and shutting the vm down again.
Wonky in a way. But also simple and robust enough. People over engineer/over think this stuff for the wrong reasons. For example, I could of course automate the provisioning of that vm. But I haven't. Because I only ever touch it once a year or so to run a quick apt-get update. I rebuilt it a few weeks ago in a different region. That was like a 20 minute job. Terraform or Ansible for vms you only create once every few years is redundant and might take more time than you would save. I can always do that when that stops being true.
I've been running this startup on the freemium layer in Github for five years now. It's great as a free service. I would actually pay for it if I needed to. I did actually pay for it before MS acquired Github in a previous startup when business usage wasn't free. But so far, there's no need for me to do that. I also run some monitoring scripts as Github actions. Simple curl jobs against our servers that trigger alerts when they fail. That has to run somewhere. It might as well be Github actions. But if/when that becomes inconvenient, I can improvise other solutions.
This sucks, it make me feel so silly after decide to move back to github self hosted runners just because I do not want to run act on a remote ARM64 server.
I was just using act (https://github.com/nektos/act) on my local server to build the X64 packages for my project, since I want to streamline it with ARM64 support, I migrated to the github self hosted runners.
This is really ridiculous, is M$ really lack that money just to schedule the Jobs running not in there infra?
solarengineer8 hours ago
[-] If Github runs the control plane, I presume there would be some costs to that. Consider the costs of hosting a control plane that assigns jobs to runners, receives and processes heart beat signals, receives log streams, exchanges file artifacts with the runner. Such actions would take up compute for the Control Plane.
Are Control Plane costs already separately charged?
Git was supposed to be "distributed", but we ended up with a central HTTP hub.
Can't we switch to something more advanced in terms of protocols (like one that always maintain 3 copies, and where people can give ressources (cpu/bandwidth/memory) in the forms of tokens)?
Will OSS (public) repositories also have to pay if they use self-hosted GitHub runners ? If yes, that seems a bit counterintuitive, given that Github hosted runners are free for public repos.
Why would a public repo use a self-hosted runner ? because the self-hosted runner storage available is only 14GB !!
My take as a cofounder of Shipfox, a company working on alternative GitHub Actions runners (same space as Depot, Blacksmith, Namespace).
The price update itself wasn't very surprising. GitHub-hosted runners historically carried a significant premium given the underlying hardware, which isn't particularly well suited for CI workloads that are often CPU-intensive. Lowering prices there makes sense and better reflects real usage.
Pricing self-hosted runners also feels logical from GitHub's perspective. Until now, GitHub Actions generated little direct revenue from self-hosted usage, despite still providing orchestration, Actions Marketplace, etc. Given how widely self-hosting is used, it's hard to imagine that remaining free forever.
For users of GitHub-hosted runners, this is clearly good news. For teams running self-hosted runners, the impact can be noticeable. For example, if your infrastructure previously achieved a per-minute cost about half of GitHub's hosted 2 vCPU rate (a conservative assumption), adding a $0.002/min fee effectively moves the total from ~$0.004 to ~$0.006 per minute, roughly a 50% increase. In setups that were much cheaper than hosted runners, the relative increase is even higher.
That said, most teams don't self-host purely to save money. Performance, hardware control, and security or compliance requirements are usually the main drivers. This change doesn't remove those benefits, but it does change the cost equation and likely forces a reassessment.
reply▲"That said, most teams don't self-host purely to save money"
I think most do. Or at least the infrastructure/compute costs are not coming from their own dept budget anymore ;)
So Microsoft is slowly killing it. Not surprising.
reply▲GitHub actions are expensive enough that self-hosted was the only real option. I can't imagine this will do anything other than push people from the entire ecosystem.
reply▲IMO it's long time coming. Streaming logs and other supporting functionality is not free. We at Cirrus Runners provide runners as a service for a fixed monthly price with unlimited usage. We target large entrerprises that save $100K+ yearly by switching to us (10-25 times). In our calculations the new per-minute fee is roughly ~0.1% of the effective per-minute cost our customers avoid by using our fixed-price model. Over providers with the traditional per-minute pricing will have bigger impact.
reply▲> Streaming logs and other supporting functionality
GitHub's log streaming also sucks. It's very laggy and chunked, whereas GitLab's is pretty much real-time.
I just convinced the team to switch to GitHub Actions self hosted for various reasons, but one of them being cost.
This is an insult to anyone who bought into GitHub. It's an insult to all of us who have been doing OSS there for years. This is how you kill your business and any loyalty or trust in your brand.
What a disaster.
Here are the practical implications and considerations to optimize for cost, given the new pricing. These are generic and ensure you think through your workflows and runners before making any changes.
1. Self-hosting runners or using WarpBuild/blacksmith runners is still cheaper
Despite the $0.002/minute self-hosted runner tax, self-hosting runners on your cloud (aws/gcp/azure/...) or using WarpBuild/... runners remains the cheaper option.
2. Prefer larger runners
If your workflow scales with the number of vCPUs, prefer larger runners. That ensures you spend fewer minutes on the runner, which reduces the GitHub self-hosted runner tax.
For example, using actions-runner-controller with heavy jobs running on 1 vcpu runners is not a good idea. Instead, prefer a 2vcpu runner (say) if it runs the job ~2x faster.
3. Prefer faster runners
All else being equal, prefer faster runners. That ensures you spend fewer minutes on the runner, which reduces the GitHub self-hosted runner tax.
For example, if you're self-hosting on aws and using a t3g.medium runner, it's better to use a t4g.medium runner since the newer generation is faster, but not much more expensive.
4. Prefer fewer shards
If you have a lot of shards for your jobs (example: tests on ~50 shards), consider reducing the number of shards and parallelizing the tests on fewer but larger runners.
5. Improve job performance
This is not new advice, but it's now more important than ever because of the additional GitHub self-hosted runner tax.
6. Use GitHub hosted runners for very short jobs
For linters and other very short jobs, it's better to use GitHub hosted runners.
Hope this helps.
Note: I'm the founder of WarpBuild. I'm biased, but the points above hold.
At getstream.io we ended up running Github Actions on Hetzner. The end-result is 4x faster builds for 3x less $$$.
Running workers ourselves was the last resort, we tried everything else but it was impossible to get fast (and consistent) build times otherwise.
In a way we are now going to get charged for Github's poor execution on Actions.
How are you guys running it? Is it via RunsOn, Ubicloud? We just moved ours to Blacksmith since I still don't want to manage the worker ourselves yet. But with this change, we might be looking into cheaper, better alternatives if there's any.
reply▲The urge to move to Codeberg grows with every passing day.
reply▲Or use Gitea or Forgejo and host it yourself.
reply▲Say I wanted to run the GitHub Action's "self hosted" runner on my own infra, then integrate it with my repo using webhooks (like I would for other CI platforms). What value would I be losing?
reply▲There are several features that are only available if you self host github runners like this concurrency issue[1] that has been open for 3 years with the only solution being to use self hosted runners. So you'd expect at least a new product release that fixes these issues before they start rug pulling people.
1 - https://github.com/orgs/community/discussions/32376
The reason this makes sense, at least for Github, is because the only valid reason to run your own action runners is compliance. And if you are doing it for compliance, price doesn't really matter. You don't really have a choice.
If you've been running your runners on your own infra for cost reasons, you're not really that interesting to the Github business.
Github runners are slow. We're using WarpBuild and they are still cheaper per-minute, even with all the changes Github has made. Then there's the fact that the machines are faster, so we are using fewer minutes.
There are multiple competitors in this space. If you are (or were) paying for Github runners for any reason, you really shouldn't be.
Thanks for the WarpBuild love!
Performance is the primary lever to pay less $0.002/min self hosting tax and we strive to provide the best performance runners.
We also use WarpBuild and very happy with the performance gain. This changes nothing except maybe it should signal to WarpBuild to start supporting other providers than Github. We are clearly entering the enshitiffication phase of Github.
reply▲thanks for the love! we are actively considering supporting other providers.
reply▲I needed arm64 workers, because x86 would take ~25 minutes to do a build.
reply▲They have these now.
reply▲Only for public repos though - if you're in an org with private repositories you don't get access to them (yet).
reply▲You do, you just have to set them up at the organization level. Windows/Linux/macOS are all available.
reply▲Maybe if everything you use is public-cloud-deployed.
Self-hosted runners help bridge the gap with on-prem servers, since you can pop a runner VM inside your infra and give it the connectivity/permissions to do deployments.
This announcement pisses me off, because it's not something related to abuse/recouping cost, since they could impose limits on free plans or whatever.
This will definitely influence me to ensure all builds/deployments are fully bash/powershell scripted without GH Action-specific steps. Actions are a bit of a dumpster fire anyway, so maybe I'll just go back to TeamCity like I used before Actions.
Not just compliance, we run CI against machines that they don’t offer, like those with big GPUs.
reply▲Performance and data locality.
You can throw tons of cores and ram locally at problems without any licensing costs.
Your data may be local, makes sense to work with it locally.
Yeah... Kind of expected GHA to be a money trap at some point. It was tempting with how easy it is to setup. And every since Claude Code integrated tightly it assumes i want pipelines in gha even though I have pipelines elsewhere. Glad I stuck with picking a different system and didn't invest a lot of time here. I had plenty of compute to run jobs myself.
reply▲steve_taylor20 hours ago
[-] So instead of addressing their runners being extremely slow to the point that a reasonable person would think it's deliberate in order to extract more billable minutes, they're charging customers for using an alternative. Makes sense.
reply▲I'm a little surprised at the outrage here. I guess sure if you're using tiny self-hosted runners this would be significant, but if you're using even an 8 vCPU machine from blacksmith for instance (16 cents per minute), this is roughly 10% extra. That seems reasonable for them providing the platform?
reply▲It’s interesting to see the posts from warpbuild, blacksmith, buikdjet and others defending their business model that was based on the inefficiency of GitHub. I love the fact that git is built in such an open way that if you are worried about running in your own infrastructure you can easily deploy it (It’s just like SSH!) yourself. At least for me, cheaper GitHub actions is a win because I can’t justify running my own git. But these companies that are based on offering you a faster or cheaper github actions service are actually the worst of both worlds: they are not your platform and they are not in the position to offer you a better service. I’m not gonna miss them when they’re gone or transformed into an AI pivot.
reply▲Probably long overdue, but per-minute price vs per-job is quite expensive. Wouldn’t like to be in the shoes of “only” 2x cheaper third parties. If they follow up with faster runners… interested to see if they ever come up with a good SDK for their scale set API, will integrate it in RunsOn!
reply▲In theory I assume you could rebuild an 'open GitHub actions' that maintained the existing API and used webhook events to trigger a workflow and github API to post status back into GitHub.
reply▲At $0.002 per minute there are at most 90 dollars in a month. Maybe even after an year of cumulative costs it's less then the cost of switching to something else. Maybe even after many and many years of cumulative costs: the larger the company the more expensive corporate inertia gets.
reply▲Our org is showing around 200-300$/mo in added fees and we are exclusively self hosting in our own on premise cluster. Kind of wild we have to pay to use our own compute.
reply▲In fairness to Github, bringing your own runners isn't "free" on their end. The orchestration happens server-side, so there is some level of cost. I don't know if that justifies the $0.002/min price - just wanted to point this out.
reply▲Oh absolutely, but honestly the self hosted runner setups that I'm familiar with are just waiting for a call. As far as I can tell GH side just routes.
reply▲if you were only paying to use your own compute, you could just use your own compute - you don't
have to use github actions, you can trigger actions on your own systems without github.
the control plane clearly has value to people beyond the compute used for running the actions, and it seems reasonable that they should charge for that if you're using it.
I agree that it’s probably not a big amount. But note that it can be potentially quiet a bit more than the 90$. Task runtimes are always rounded up to the nearest minute.
For example, in our pipeline we have 5 different linter tasks (for different subprojects), running each only a few seconds. Nonetheless, we’ll get billed for 5 minutes on every commit.
Ah I see, they are not minutes as on the clock. They are runtime minutes. That changes my assessment. I was thinking that they picked a balanced price point not to scare away many people except probably personal projects or unfunded open source. If it's something potentially in the ballpark of $500 per month it's a bit too greedy. It's more like: we want only corporate customers, free tier users need not apply.
reply▲Per minute per runner. If you have multiple workfows/jobs running, it can add up.
reply▲We are a ~20 person team who use private runners and this will increase our annual costs by ~12k/yr. This is a huge relative cost increase for us. If anything this hurts small teams that focused on expansive automated testing more than giant orgs.
reply▲The funny thing is if GitHub let me pay extra for an actions runner that was not a potato, I would happily give them so much money. Instead they want to penalize me for working around their broken product.
reply▲Charging by minute might push people toward shorter, noisier and more fragmented pipelines. It feels more like a lever to discourage selfhosting over time.
It's not outrageous money today, but it's a clear signal about where they want CI to live.
Github, thanks for making this service available free for public repos, it's a big boon.
Currently the runner has only 14GB of disk space, if that can be made to 50 GB, that would be amazing !
reply▲QuiCasseRien23 hours ago
[-] More than 6 years users of OneDev (onedev.io).
- Git repo
- Ticketing, Kaban
- Full helpdesk
- Complete and full CI/CD
- everything links via custom workflow
- self hosted
I still dont know why everyone hasn't switch yet to that banger.
I really wanted to like it but the UI always put me off. Also tending to prefer a more open development model these days. Thankfully at least for dev gitea and forgejo have both come a long way and the CI is pretty decent now (though they still dont have a gui workflow builder!).
reply▲Charging for the self hosted runners - That's close to flat $90 per month for a machine that you host yourself no matter how small or large the machine is.
reply▲this is the third article about it, we know, good times are over, will start migrating towards something else
reply▲It definitely adds to frustration for some people; this can not be denied.
reply▲don’t lie you’ll just bitch and moan and keep using it anyway
reply▲sallveburrpi20 hours ago
[-] the downvotes are from those who can’t cope with this truth
reply▲Our org just migrated from Bitrise to self-hosted GHA runners just a couple of months ago, with cost savings as a main reason. I already foresee an interesting conversation coming up tomorrow.
reply▲$LLM spinup a jenkins cluster on my qa infrastructure please
reply▲So they are finally doing this. Our github account rep mentioned this back in February, but then they kept postponing and heard nothing so I was hoping they realized how stupid this idea was and abandoned it.
My org sadly has a lot of github actions workflows, even after this it's not expensive enough to justify migrating away, but with all their downtime and bugs they are really pushing us closer and closer.
This is a serious issue. How is it possible for GitHub/Microsoft to charge me for using my own machine as a self-hosted GitHub Actions runner?
reply▲They're charging you for orchestration, log storage, artifact storage, continued development of the runner binary itself and features available to self-hosted machines. What would your own machine do without the runner and service it connects to?
reply▲I'll be investigating gitlab tomorrow
reply▲Have used all of the big 4 forges in anger over the last decade. GitLab isn't perfect, but I'd take it over GitHub any day of the week.
reply▲it charges you to use the platform features that enable your use of self-hosted runners
reply▲For the same reason they charge you for running Word, even though you're the one who has to write, I guess?
reply▲hoping for some disruption here. gha is an absolutely horrid platform for anyone trying to build optimized workflows. so many bugs / rough edges that haven't been addressed for years, the hosted runners feel like decade old compute. missing all of the modern features (like dynamic pipelines) other providers offer.
to top it all off, they round up to the nearest whole minute instead of billing for actual usage which i assume they'll use for this new charge.
Would also be interested in a better platform
Earthly did not work out, and dagger had the problem of we support everything but but nothing is great
Ah the monoculture comes back to haunt people. Who could have seen that one coming?
reply▲I understand that orchestration,log storage, keeping software updated can cost money, which they seem to recover from charging for software. Hopefully there is support now included with self hosted runners being charged.
reply▲If they charge me for my self-hosted runner i will just move to Gitlab. This is theft..or let's say this is microsoft.
reply▲I would remind everyone that lots of free solutions like Forgejo exist with much better security posture.
reply▲I assume they want us to pay for their orchestration and also push customers back to using their compute so everything is stickier.
But nothing they've done in the last few years has demonstrated improvement in this area. As the person with both purchasing and final authority on these things in my org, it's hard to stomach.
voganmother4218 hours ago
[-] With their availability issues it will be hard to forecast costs of “continuous” operation. I guess everyone using ARC can get rekt, why would you put in the work to move to their next bs when you can just leave?
reply▲Geez. This would've been much more agreeable had they bothered to fix years-old open bugs with self-hosted runners
reply▲I'm not sure what response they expected, but for some reason it makes me think not that.
reply▲How will this hit OSS projects which rely heavily on github actions? I’m thinking of projects like nixpkgs, which is the backbone of nixos and always has dozens of actions queued or running. (I am using nix as an example for scale, but I am not involved in the project and my description might be inaccurate. I’m also not familiar with nix’s financials at all.)
reply▲> Standard GitHub-hosted or self-hosted runner usage on public repositories will remain free. GitHub Enterprise Server pricing is not impacted by this change.
reply▲Are there bring-your-own-agent CI platforms that don't have pricing structures like this? Buildkite and CircleCI do.
reply▲CircleCI does only charge for self-hosted runners generated egress and/or artifact storage:
"Any Network Egress to CircleCI will be charged. At this current time, this includes CircleCI Caches, Workspaces, and Artifacts and will be charged at the normal rate according to your Usage Controls.
The only network traffic that will result in billing is accrued through restoring caches and workspaces, and downloading artifacts to self-hosted runners. Retention of artifacts, workspace, and cache objects will result in billing for storage usage.
Since your builds will not be running on CircleCI's Infrastructure, you will not be charged compute credits"
https://support.circleci.com/hc/en-us/articles/2064321965685...
I think that's fair. In my personal opinion most people started using GitHub Actions because it “came for free with the VCS and/or our MS contract” and it was “good enough for the job”. Now might be a good time to look around at the alternatives again. There is a reason that f.e. CircleCI is doing fully focused CI/CD for 10+ years and is still going strong. Plenty of businesses don’t want to put all their eggs in one (MS) basket, for all kinds of reasons. I guess today one of these reasons became obvious.
Disclaimer: I work at CircleCI.
CircleCI charges for concurrent job runs (which include self-hosted runs), no? They (you, I guess) obfuscate that by saying you get "Unlimited" if you take the "Talk to sales" route but that's not the same as not charging.
reply▲It's not obfuscated. The free plan gives you a max of 5 concurrent self-hosted runners. If you need more you can upgrade your plan:
https://circleci.com/pricing/#comparison-tableThere simply is no free lunch, somewhere someone needs to spend effort and time on managing the orchestration layer for the runners, and there is also network traffic and storage in play that costs money. If you need a future-proof CI/CD platform, it takes some investment. I agree that the Github "pay per minute" approach doesn't feel right, most people would probably find a "pay per orchestration job" or something more acceptable.
Anyway, there are alternatives out there :)
Agreed there's no free lunch, GH is moving from more generous than the industry to as-generous (or less-generous depending on your opinion of per-minute versus per-job).
reply▲By default free plans can run 5x concurrently on self-hosted, 20x minimum for all paying customers, and yes there's a "talk to sales" for >20x on the pricing page
reply▲> > We are introducing a $0.002 per-minute Actions cloud platform charge for all Actions workflows across GitHub-hosted and self-hosted runners.
Holy s***
That's more expensive than an m8i.large.
What on earth.
That's actually $90/month, that's kind of crazy.
I realise 100% utilisation isn't realistic, but that still sounds very expensive when you're already BYOB.
>
I realise 100% utilisation isn't realisticIt's worse than unrealistic. It's ludicrous. Any company running more than an hour of actions workflows per week on GitHub can afford a few dollars a month for infrastructure. The per-minute charge is less than the cost of a millisecond of engineering labor time.
Monorepo, though Gitlab, self-hosted runner, 41 hours in the last week.
reply▲Tell me that $20/month is a notable amount of expense for your business spending 41 hours per week on workflows. Go on.
reply▲Dude why are you so determined to defend this pricing change? You're all over this thread arguing with people that it's not a big deal. If it's a big deal to them, why do you give a shit? It's not like it's your problem if people take their business elsewhere for a poor reason.
reply▲This increases our CI compute costs by 30%.
reply▲We'd be at roughly $700/month at current usage.
reply▲Per-minute pricing for self-hosted runners seems like a very fast way for them to force everyone who actually is using self-hosted runners to migrate away.
I suspect we'll be doing that sometime in January or February.
I guess forgejo is the easiest migration path? https://forgejo.org/
everyflavourvms19 hours ago
[-] I haven't used Actions in a professional context so am just wondering (and this might help coming up with arguments should $c-suite start requiring a move): is a "runner" equivalent to an executor slot in Jenkins? As an example, we currently have some builders with 20 executor slots and they might all be orchestrating test runs in parallel (these do not consume much CPU as all they are doing is instructing _other_ VMs, created on the fly, to do the actual work). Would that count as 20 runners in Github Actions, hence costing $0.002/minute times 20?
reply▲paulryanrogers19 hours ago
[-] Each GHA runner gets its own VM. So every minute those are running you're billed. The runners do work independently which can save wall time.
reply▲This customer will be leaving GitHub action runners for punishing self-hosting.
GitLab CI and others seem to be perfectly serviceable.
Maybe it's time to start dusting off the ol' Jenkins-fu?
Charging per minute for self-hosted runners seems absolutely bananas!
PLEASE stop propping up the narrative that the GitHub Actions control plane was previously free. It never was. Pricing is not that simple. I see way too many people in this thread, and even GitHub Actions competitors promoting this nonsensical narrative.
reply▲Ahh, so since GitHub is completely incompetent when it comes to managing a CI they are going to make it worse for everyone to get their cut.
I hate GH Action runners with a passion. They are slow, overpriced, and clearly held together with duct tape and chewing gum. WarpBuild, on the other hand, was a breeze to setup and provided faster runners and lower prices.
This is a really shitty move.
Hey GitHub, your Microsoft is showing...
I have never been a fan of GitHub and their entire system, always felt Bitbucket or GitLab were superior in terms of the tooling and included features across all plans.
However, my experience with GitHub Actions was really poor. Some build that would run perfectly on my local machine and any other servers we have hosted would always time out on GitHub runners. I went back and forth from small runners to large runners and the result was always the same. Then I found that there are third-party companies just offering replacement runners for GitHub Actions at less than half the price for an amazing reliability and cost. It was a night and day difference.
Now... this move by GitHub is almost unbelievable. Charging folks to use their own machines
Hey - thanks for the WarpBuild love!
Given github ran 11.5 billion mins of actions in 2025, and most of them would've been on self-hosted runners, this move makes some sense from their POV.
However, this is still an... interesting... move, especially after bitbucket got all that hate a few weeks ago for doing something similar.
ed_blackburn22 hours ago
[-] Microsoft are really sweating GitHub now aren't they? It wouldn't be so bad if it improving but there is certainly a perception that it is costing more for a poorer product, irrespective of the new features they're layering on.
reply▲AWS code (build|deploy) supports GitHub actions workflow, gitlab does, gitea (codeberg, forgejo) too
The biggest issue is the compatibility, forgejo doesn’t have all the actions available that GitHub does nor some of the same functionality
it explains github actions update better than github
reply▲Given that I can dump hundreds of TBs into the private container registry without paying anything I'm pretty surprised that they now charge for what is basically providing log streaming and retention.
reply▲october814016 hours ago
[-] It's effectively proven at this point that any good product that is run by a publicly traded company will turn to shit.
reply▲let us open a petition to urge M$ to also charge us for git commands:
- git clone: 0.10$
- git commit: 0.001$ * number of files
- git pull: 0.01$
- git push: 0.0175$ * numbers of commits * number of files
- git merge: 1.25$
- git merge --squash: 2.00$
a nice feature would be if they limit the number of branches, too:
- <=2 branches: free
- <=5 branches: 3.00$ per user per month
- >5 branches: contact enterprise sales
It's a bit weird, they add pricing for this, but reducwle GitHub-hosted runners by "up to 39%".
Not sure about the "up to" implications, but I guess it's just Microsoft trying to make github a bit more freemium tm
The full quote:
> And we’re reducing the net cost of GitHub-hosted runners by up to 39%, depending on which machine type is used.
> The price reduction you will see in your account depends on the types of machines that you use most frequently – smaller runners will have a smaller relative price reduction, larger runners will see a larger relative reduction.
I've been running self hosted runners for my company using Actions Runner Controller (ARC) on my own kubernetes infrastructure. Could never really get the devs invested in GitOps style dev cycles so I may just chuck actions and use a more nightly or on demand style build server since that seems to be what they desire and expect. I always expected this day to come so my actions use very little github/actions specific stuff, mainly they just kick off scripts already. I do wonder how hard it would be to create my own github API pollers etc but not sure I want to invest any further in anything github specific. Good news is the effective date is March and the initial prices for my usage will probably be very low but I fully expect them to push further price increases / monetization / lock-in.
reply▲We all knew this would happen. For open source projects one step local build and test is superior to full automation for this reason. It lasts forever whereas these automated server configs require ongoing maintenance.
reply▲But open source is still free for now.
reply▲Anything that prices spammers out of abusing GitHub actions is a win in my book...
reply▲Maybe it's a lack of imagination on my part, but how do spammers abuse self-hosted runners?
reply▲Form submission spam. Unique/'untraceable' IPs...
reply▲How do they abuse self hosted runners?
reply▲ZeroConcerns14 hours ago
[-] Malware in build scripts/dependencies. That's not exclusively credential/crypto-stealers, there's apparently also a healthy demand for various types of spam straight from corpo gateways...
reply▲almosthere21 hours ago
[-] Well sounds like $40 per month more for us. Looked at CircleCI pricing, and mostly because of HOW they charge, it would be $3000, so Github it is.
reply▲Is that because you have loads of users? (curious CircleCI employee here)
reply▲almosthere21 hours ago
[-] Your pricing page seems to have changed intra-day. but now it's about $400ish.
30 users + 500 builds.
However I don't know what counts as a build, since a typical commit to an open PR uses 10 GH runner machines simultaneously doing odd jobs like integration tests, releases, deploys, etc...
Can you send a link to the page you’re looking at? Thanks!
Pricing should mostly just be users + build minutes (for cloud runners) + storage. There is a few other optional, feature specific costs. Self hosted runners are free, but you need to self host caches/workspaces - our native ones have an egress bill to self hosted runners.
almosthere21 hours ago
[-] https://circleci.com/pricing/build-your-plan/If self-hosted runners are free that would change our equation a bit. I'll talk to some folks here, I liked using this product at another company I worked at - but this would most likely shake out AFTER Github charges us the first time.
reply▲Good to know - and I can see the confusion on that page - I'll pass on the feedback, thanks!
reply▲Surely self-hosted runners are a retention mechanism with a relatively low cost for GitHub? How do they rationalise the long-term harm that this causes over just swallowing the relatively small amount it costs to keep customers paying?
People, now, are going to be annoyed and/or pissed off about this and look for/move to alternatives. It's not even that difficult to move, and if you're already self-hosting runners you're also in the position to self-host your Forge or move elsewhere.
Actions isn't even good enough to demand this. They're slow, buggy, and full of shit.
Feels super much like the classic Microsoft short-sighted bullshit. Take something that's been running well, and people were happy, then abruptly change it in disruptive ways and slowly kill your products that were doing just fine.
Github can just drop Actions pricing and leave the self-hosted stuff alone, and people would even have extended more goodwill. Is MS this short-sighted and greedy as to push further toward killing a golden goose?
molszanski20 hours ago
[-] Is it a runner minute or workflow running minute? That would be a massive difference. Would people pay for idle time or not?
reply▲Back to Buildkite I go.
reply▲Isn't it like way more expensive and restricted? They were very competitive in the early days, but currently they are more capped than anything else it seems. Especially for self hosting..
> Hosted Agents
>
> 2,000 minutes/month
:-o
Buildkite doesn't have per-minute charges for self-hosted agents.
reply▲Ok, they have changed their pricing. Currently they are capping the number of concurrent agents. At one point, they introduced minutes cap and that was very big step down.
reply▲Hmm... News about massive RAM price hikes. Then GitHub decides to charge for per-minute. Do they keep a lot of stuff in RAM while a workflow is running?
reply▲what are the opensource alternatives to selfhosted runners ?
reply▲That's just the point. Selfhosted runners were the alternative. The only alternative for "runners" is Github-hosted, 200%-markup runners.
Now the only alternative is to move builds, CI, etc. off of GitHub's platform entirely, and maybe your source control as well. In other words, a big pain. Github seems to have entered peak encrapification: the point where they openly acknowledge rent-seeking as their product approach, fully deprecating "building the best, most reliable, trustworthy product." Now it's just "Pay us high margins because the effort to migrate off is big and will take too long to break even."
Basically the modern day Heroku business model.
Good project. Picked up the pieces when Drone CI rug yanked.
reply▲Genuinely curious about this as well. It's a major bummer that self-hosted infra can't be used to validate GitHub Pull Requests now; basically means I'll have to move my entire workflow off of GitHub so that everything can be integrated reasonably again.
reply▲That's not exactly true. You just won't be able to use self-hosted infra to validate GitHub PRs a) using GHA and b) for free.
GitHub still supports e.g. PR checks that originate from other systems. We had PR checks before GHA and it's easy enough to go back to that. Jenkins has some stuff built in or you can make some simple API calls.
It's not as convenient, but it works just fine.
Won't those other systems create a cost that is metered in terms of run-time?
reply▲I suppose any compute resources would, but it wouldn’t be GitHub charging you for it if you’re not using GHA.
reply▲We use a combination of GHA and Jenkins jobs. All these end up as checks on GitHub. You could then proceed to say "Allow this PR Merge button if check A is ok and check B is ok" where check A arrived from GHA and check B from a Jenkins job.
reply▲Gitea + Gitea Actions works approximately as well as GHA. For GitHub specifically, you're back to setting PR checks + commit status programmatically through the API.
reply▲Don’t forget the windows tax!
When building on self-hosted windows machines, you actually pay three times.
Oh I wish I could make my customers pay three times for everything I deliver, I might be as rich as Bill by now.
Atlassian recently did this with BitBucket self hosted runners. Is there a CI/CD cartel or something?
reply▲I guess Jenkins gets back in the game.
reply▲If I have a VPS, what should I be running on it to replace github actions? (eg run tests, return pass/fail to github PR)
reply▲Makes you wonder, how much the AI madness will be able to cannibalize other buisness sectors before it encounters the limits of growth there, leaving behind hollowed out eco-systems - similar to how adds ruined everything.
reply▲I do not even understand why any decent size eng org uses actions. It only has rough edges.
reply▲NamlchakKhandro13 hours ago
[-] Someone in this thread will unironcally suggest Jenkins, CircleCI or Bitbucket.
These people will forever unto the end of time into their afterlife have a Harem of old ladies following them around laughing at their never ending hilarious hot takes.
Microsoft has mierdas touch.
reply▲reply▲I learned this lesson the hard way recently. I'm on the 50k minutes plan and burned through my entire monthly allowance plus a $100 overage budget in about two weeks.
My mistake was a combination of triggering workflows on every push, using macOS runners (which I didn't realize had a 10x multiplier compared to Linux), and forgetting to set aggressive timeouts.
I'm sharing this because the support experience was actually the highlight. I opened a ticket expecting to just eat the cost, but they sent a detailed breakdown of my usage/mistakes the next day. Even though it was 100% my error (tho I used to think macOS runners were only 3x? True, did that change? anyway), they gave me a $50 coupon to offset the overage. Amidst the pricing discussions, I think it's worth noting that their support team is still very human and responsive.
Is there a great opensource CI system that integrates nicely with github repos?
reply▲Use Blacksmith. I promise you won't regret it.
reply▲Wild to see that they make you pay an expensive price to use your own hardware...
First, they are free quota and the free self hosted runners to kill the previously existing competitors by dumping their price very hard, then, once alternatives are already dead, they can start to take their margin. Disgusting!
reply▲nwellinghoff15 hours ago
[-] What a fucking joke. They are going to charge me for running a script I wrote on MY server that is merely launched by their server that I am already paying an outrageous amount for to have a private repository. By the minute!!!! It never ends.
reply▲Founder of Depot[0] here. I'm disappointed by this change and by the impact this is going to have on all self-hosted runner customers, not just us. In my view, this is GitHub extracting more revenue from the ecosystem for a service that is slow, unreliable, and that GitHub has openly not invested in.
We will continue to do our best to provide the fastest GHA runners and keep them cheaper than GitHub-hosted runners.
[0] https://depot.dev
Love how they drop this news right before everyone goes away for the xmas holidays and it kicks in right as you come back. Or before you come back if you live outside the US.
reply▲I guess this is on brand for Microsoft. Just lame to go through the trouble to self host runners and still get tacked on with fees after the fact.
Hard for me to feel like our industry is innovating and not just gouging with the rest in the battle for enshittification.
I will intentionally start exploring other options even if the cost isn't high, because I don't want to support this type of thing.
I have a love-hate relationship with GitHub Actions. Love because they are right there in my GitHub repository. Hate because they are very brittle once you move out of the happy path.
It seems GitLab has a much better experience in this department, but their pricing is hard to justify for us...
Genuinely curious if folks here had better experiences or recommendations for a smooth CI/CD experience.
Love-hate for me as well. Love that there is native github integration for triggering events and other github bits. Hate the brittleness and anymore the reliability even if you are just using the control plane. I've always sought to keep my actions as mainly just calling existing scripts, that is keep logic out of them and make them relatively dumb wrappers but it would still be some effort to get off it.
reply▲So can we just go back to using external CI platforms that just interact with GitHub's commit status API or whatever?
reply▲timetraveller2616 hours ago
[-] We've been using woodpecker-ci for the last two years, it's really simple to setup and maintain for anyone looking for a self-hosted ci solution.
reply▲i think it's time to migrate like zig.
reply▲Use open source software. Buy your own compute. Make the effort. It's worth it.
reply▲I'm kind of ok with renting compute so long as it's running open source software.
Basically I'll gladly pay for a service, but I don't like getting locked into that service. If the payed service is using FOSS, I will always have the option to migrate if the provider starts to misbehave
systemBuilder22 hours ago
[-] $3 a day, $100/mo to run your own github actions (which is a programming language based atop json ... sheesh). Ugh!!
reply▲it looks ms wants to kill all their IP, xbox, windows, now github
reply▲blanket 30% profit margin is great right?
reply▲I read this and I'm thinking I should just get Claude to write me my own GitHub (with blackjack! and... nevermind)
I'm in the era of writing my own tools, not to share just for me or whatever group I'm working in. If you're going to charge me for something rife with annoying struggles, I might as well be annoyed by a tool I control.
ah! a fellow futurama lover, i see you
reply▲Is there any included free amount of platform minutes for private orgs/repos? Currently using Blakcksmith with arm64 and do around 600 minutes a month (very small). I get 2,000 free minutes of GitHub runner time for free, so maybe have to switch to using GitHub native arm64 runners.
That being said even with no free platform minutes my Blacksmith usage will only $1.20 a month in platform fees, so inconsequential.
I was worried about this, but $10/mo for 5000 self-hosted minutes isn't terrible, the self-hosted runner feature is great for our use cases where the repo is too big to run in the cloud generally and/or ingress/egress is too expensive.
reply▲Why are there no changes for plans with included minutes e.g. enterprise that has 50000 since the runners are now cheaper? So now the included tier has effectively been reduced.
reply▲Do private orgs/repos get any free platform minutes? Currently I’m getting 2,000 free minutes of action runtime with a private org/repos.
reply▲how long before they start skimming OSS projects that are public but nonetheless have Github Sponsors income. I mean that's money right there for them right
reply▲Wasn't that the key concern of Zig moving off GitHub?
reply▲dunno, but this is only actions. You can use github without being dependent on actions.
reply▲Oh great. I finally get used to GitHub Actions after Travis CI shat the bed, and now I have to find something else.
Thanks, enshittification.
Hey man, that's not fair. They cannot enshittify what has always been shit to begin with.
reply▲Oh you sweet summer child
reply▲What part of this is “enshittification”? It’s just a company starting to charge for a formerly free service. Hardly seems like that aggressive a move.
reply▲They're squeezing their customers after locking in to juice their margins, having become a monopoly/monopsony. This is the classic enshitificaton playbook.
reply▲matthewmacleod22 hours ago
[-] Nobody is locked in (unless they made some incredibly bad decisions) and this is a tiny fee in exchange for a useful service. I’m just baffled by the response to this.
reply▲It's not baffling if you read his Enshitification book. This is phase 2.
In 2010, people were saying it was very reasonable to start prioritizing publishers' ability to reach you over your organic contacts. After all, Facebook is providing this utility for free; shouldn't they be able to extract some additional revenue from their platform? And here we are in 2025...
From
https://www.wired.com/story/tiktok-platforms-cory-doctorow/"Here is how platforms die: First, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die."
We are on step 2: then they abuse their users to make things better for their business customers.
matthewmacleod22 hours ago
[-] It is not abuse to charge what amounts to a relatively small fee for a useful service.
reply▲some_furry10 hours ago
[-] It's not "a relatively small fee for a useful service".
It's an unnecessary fee to use self-hosted (i.e., not GitHub-hosted) components in CI pipelines.
> TL;DR GitHub is adding a $0.002-per-minute fee on all GitHub Actions usage, so the control plane is no longer free.
That's not true for _all GitHub Actions usage_.
https://resources.github.com/actions/2026-pricing-changes-fo...
> Standard GitHub-hosted or self-hosted runner usage on public repositories will remain free.
That was my biggest concern... I've used runners for personal/public repos because they're there and generally good enough. If I were paying for it, I might be inclined to look at faster options.
reply▲TLDR: Github is no longer free for self-hosted actions in private repositories, although there is a free quota.
reply▲Everyone in this thread has gone absolutely insane. $5/month gets you 41 fucking _hours_ of continuous operation. If you're not utterly abusing the platform, this falls extremely below the threshold of caring. And if not, what the fuck are you even doing with all those hours? The new per-minute charge is less than one millisecond of engineer labor cost.
reply▲I have a nightly software build of a piece of software that takes 6 hours to create a 70GB artifact. The build process requires a GPU so it runs on my own HW. That's ~180 hours per month for this job alone. Is that really so hard to imagine?
reply▲I don't know how much of that 6 hours build is tangled up in github workflows, but if it's a single contiguous block, you probably could make it near zero by making the self-hosted runner do only the preparation and only the final upload process (workflow_dispatch when the build is complete).
reply▲Most of it is just time waiting either while the source assets are downloaded (I clean slate it, that's the point of CI after all), the build itself runs, or the artifact is uploading to it's storage home. I'm sure it could be re-architected to use less actions minutes but if I'm going to redo it I will probably just move away from actions altogether because it's only loosely linked to Github anyway (runs on a schedule) and that way I am insulated from any future changes they come up with. The hardest part will likely be figuring out the Slack bot posting, I do use the marketplace action for that, but that's probably low lift. With LLM assisted coding I'm leaning more and more to little in house apps for stuff like this, it keeps you from dealing with lock in and other extractive gotchas.
reply▲Slack posting is literally one-line curl with a token. That's what the fancy marketplace action does behind the scenes.
reply▲Yep and a good example of how using the convenient best practice pushed by the vendor (the marketplace action) isn't a good idea. But I did.
reply▲I think you significantly underestimate the number of CI minutes people are using in practice.
(Which, yes, has implications for energy use/climate change too for sure).
It doesn't look like i currently have access to the usage data on any of the lots-of-runners-lots-of-PRs projects I currently work on (which are still probably way less than some large companies).
>
some large companiesAny "large companies" don't give a shit about things at this cost level. They spend more on the time it takes you to open the door. The number of CI minutes could be astronomical and it still wouldn't rate above the threshold of caring. The time people in this thread have spent wringing their hands is way more expensive.
per minute billing is hard to wrap around the head
On my larger organization, we have on average 20 to 30 *active* runners during business hours. Assuming 5 on the off-hours, my napkin math says it comes down to about 10 fully-utilized-runners per month, so about 864$/mo. For the size of my organization that is honestly totally acceptable.
This is assuming 0.002$ per minute of job being actively executed. If it turns out to be 0.002$ per minute of *runner being registered* on the control plane, it would increase quite a bit. We are still using the old HorizontalRunnerAutoscaler with actions-runner-controller, with quite a pool of prewarmed runners idling to pick up a job. It would be a strong reason to use the new RunnerScaleSet (to take advantage of the reactive webhook-based scaling) and keep a very lean pool of prewarmed runners.
TheCondor42 minutes ago
[-] We have the same question, our runners are registers 24x7 but we probably only use a few hours a week.
I get the logic of it, they have to have some sort of task running on their side when the runner is working. If it's only build time, then we don't care.
It could be 1 dollar a month, I'm still not paying to use my own ressources
reply▲jrochkind121 hours ago
[-] Well, you obviously are using their resources, to kick off and register statuses of the jobs running on your resources, right? That is probably worth $1/month to you?
reply▲Doesn't this depend a lot on how long your actions run? Like, you may have already invested in your own hardware (maybe because your actions use a lot of resources and it's cheaper) and now you have to pay per-minute of action runtime for the API that does the bookkeeping?
reply▲$3 gets me 730 hours of comparable Vultr VPS time.
reply