From Terminal-bench-2.1 details,
> We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.
This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.
For reference, in tbench-2.1,
1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)
2. 8 out of 89 tasks allow 8GB RAM
This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1
As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.
I thought the benchmark was supposed to be about terminal use and specifically chaining together lots of bash tool calls. Which test cases does this matter for?
A few examples from memory:
1. This task [1] asks the agent to train a CNN under 1 CPU, 2GB RAM, 10GB storage. If you allow high resources, weaker models often succeed (the most clock time actually goes in waiting for the network to train).
2. This task [2] asks agents to implement a complete MIPS interpreter in JavaScript in 1 cpu and 2GB RAM. A common failure mode is OOM, at least in the earlier buggy versions that models run to get feedback. When OOM hits, the task is killed, no do-overs.
3. A lot of tasks involve building projects with a single core supplied. If you use -j12 type options, it will actually be _slower_ to build and the task will more likely miss the timeout. Having more threads squeezes the end to end time. This is a big one actually since the most common failure mode (from what I have seen) is the task timeout hitting before the agent finishes
[1] https://github.com/harbor-framework/terminal-bench-2-1/blob/...
[2] https://github.com/harbor-framework/terminal-bench-2-1/tree/...
- Chinese models
- Grok
- Meta
- OpenAI
- Anthropic
I think this is a win. I'm building like crazy to take advantage of all these subsidized tokens while I can.
On the one hand, because it is easy to build products, more and more people will build. And more and more products and features will be built. However, a lot of people who are non-technical will also try to build, but they get stuck, and then they will need engineers. The sheer volume of product built by both experienced technical companies and non-technical novice startups and founders and wannabe founders is going to be massive. That is the bull case for having more software engineers needed in the near future.
On the other hand, in a year or so, people will build all these products, and most of them won't be able to market them, sell them and make money. Eventually, there won't really be a need for that many software engineers.
I think overall the bull case is probably going to win net net.
I’m “seeing” more devs stepping into the SendCutSend stage where they’re cleaning up/fixing/productizing vibe coded projects so maybe there will be some new demand in that space?
Another example is when the WWW first became available, and suddenly everyone COULD be a publisher (browsers even included built-in HTML editors), and for a while MySpace pages proliferated until the excitement died down and people went back to being media consumers.
I expect we'll see the same thing with consumer use of generative AI. Suddendly everyone is generating 3-D worlds/games with Fable because they can, but I expect that just as with the web the novelty will wear off and they'll leave it up to the pros.
Professional use of GenAI, and coding in particular, is certainly here to stay, but it seems we're still in the early experimental/hype phase. At least tokenmaxxing has passed, and it seems most companies are now paying attention to, and limiting, how much they are spending, but it doesn't seem we've yet progressed to the stage where companies are paying attention to what they are actually getting out of it - is the money spent showing up on the bottom line in the form of increased revenues.
And those people won't need to be software engineers.
> but they get stuck, and then they will need engineers
You've implicitly assumed here that the AI systems will always be worse than the average engineer. That is IMO myopic. I'm not sure that it's even true now let alone in the nebulous future.
Most of what we do as engineers is precisely describe or analyze the behavior we want or the behavior we don't want. All other engineering skills that are useful are ultimately downstream from understanding the behavior of software enough to know which parts to keep, improve, or jettison. Chatbots can take care, somewhat, of analysis or expansion of instructions.... but they can't read minds. I don't see that changing any time soon.
I don't know who needs to hear this, but neither can humans.
You've implicitly assumed here that AI systems will always be worse at contextualizing and framing questions than the average engineer. I'm not sure that it's even true now let alone in the nebulous future.
You haven't narrowed the fundamental myopia of the assumption here, just dressed it in slightly different clothing.
How would they know what to ask or contextualize if they don't know what the user wants?
> How would they know
How would you? The answer is the same.
If you're implying chatbots can ask their "client" what to build, good luck with that—contractors are at least liable for what they produce and have extreme incentives to ensure that their clients are happy. To the extent of refusing to build anything if they don't know what they want....
Plus all the other things that software engineers generally have not learned to a professional level even if they picked up the basics on the job by osmosis, because figuring out the customer's needs (and what they'll pay you for which may be different) is the job of a business analyst, a PM, or a UX researcher, and those are different skills and two of them may come with a Business Informatics degree rather than a CompSci one.
LLMs can be "eh, better than nothing" at many things, not just code.
Plus, this is software "Engineering" we're talking about, which famously gets scare quotes in comparison to all the other forms of engineering because unlike them we don't have as standard things like professional liability insurance to cover serious professional errors of judgment the way someone who signs off on a bridge that collapses would have.
I think for a lot of type of software we have now reached peak employment.
Someone payed a few k just for a normal website.
Do you think that Chinese software industry is that relevant to the kind of software market talked about on HN? I.e. lots of enterprise b2b and infra companies.
Chinese companies have always had a very low willingness to pay for software which kinda breaks the flywheel of B2B SaaS companies and companies to service those companies all the way down.
Are we still left with this mindset? Maybe once upon a time but it has definitely been changing.
There's plenty of B2B and enterprise SaaS companies in China serving the Chinese market. Maybe not as many, but no longer the very low of the past.
I also would not say enterprise were not willing to pay, even many years ago. It's the SME that refused to pay. Large CRM, ERPs etc have always existed.
- DeepSeek
- GLM (Z.ai)
- Minimax
- Kimi (Moonshot)
- Hy3 (Tencent)
- Qwen (Alibaba)
(Each one of these with weights available to download and run locally)
I do not know if competition is good, we will see in a few years.
Looking forward having a physical job for a change :D
I do not mean Suckerberg or Eric Schmidt.
No wonder we still can’t get climate change under control
This is was historically a money issue, being green used to be wildly more expensive.
Now being green is cheaper, the limiting factor is how fast PV and batteries can be made or imported.
Recent reports of the sum of all US data centres currently in planning, has a power demand exceeding the (capacity-factor-adjusted!) global annual supply of new PV.
This would be less of a problem, but still a problem, if Trump wasn't trying to get in the way of anything green, or if the companies building data centres decided to also support factories to make more PV.
* Planned new demand: 300 GW; PV factory capacity ~ 600 GW nameplate, but the capacity factor is 14% so that's really 84 GW on average.
https://dev.meta.ai/docs/getting-started/pricing-rate-limits
If they have a really good model, it makes sense to subsidise it, to gain users, before they align prices with competitors.
I really dont see how anyone's willing spend more than $1.50 per mm output. Let alone $15-50. Does anyone actually pay for usage based billing as a consumer?
He doesn't have to match Anthropic or OpenAI revenue if he can deflate theirs by 99%.
All he has to do is keep spending a few billion dollars developing frontier models, release them as open weights, and turn coding models into a commodity. Very few people are in a position to do this and for it to make business sense.
That's quite likely where things are headed regardless, and he could speed it up significantly.
We should all hope models move from proprietary products to commodities the way compilers did.
Not the way you're implying?
The GLM 5.2 hype was blowing way before this. Neither xAI nor Meta have really made a difference in a different way - similar results / similar pricing (to GLM 5.2).
To be fair, seems more correct to compare against similar strength models if your main edge is pricing.
What kind of use case would be best for that shape?
Bug diagnostics is about being okay at coding but better at tooling.
Given a good diagnostic report, it can be handed to opus for the fix.
Opus is okay at writing reports, but it still regularly gets table widths wrong in typst documents, leaving the last column full of text but only a handful of characters wide.
I see models fail on tool calls that involve API requests to a specific API, internal or cloned Makefile calls, npm run commands, etc.
:(
Well, Vietnam is not in the list of restricted territories.
Anyway, what is "your region" ?
Is this where I am now, or is it where I activated my Oculus 2 five years ago ?
I don't know where I need to sign up to try it out. What is pricing? Is it API or subscription, what?
I had the exact same experience with Grok 4.5 as well.
I for one am really glad to get competitive models that will push the major labs to bring prices down. While Chinese open source labs are also great, unfortunately when it comes to US/Western political pressure it won't often have as much of a bearing on labs bringing prices down, especially for enterprises.
Also if these numbers are true, this is truly breaking ground finally for Meta.
I have questions regarding if I should even care but I don't so Meta please keep enjoying the irrelevance. lmao
I'm going to assume the only "region" that's permitted is the USA.
Meta's AI org when from a total mismanaged dumpster fire for multiple years to delivering a competitive model in less than a year on essentially their first try?
so yeah, this is essentially their first try with a completely new org.