This latest crop of frontier models (Fable, Sol) are very good at broad exploratory directions like "audit for bugs" or "find places in this codebase that I custom-implemented functionality that has since become available in the libraries I am using" in ways that prior generations would fail or not return as high quality results.

I did a recent head-to-head comparison between Qwen 3.6 35B running locally and Fable on the above and while Qwen did an admirable job finding a good handful of things, Fable returned many more, and more serious, issues. This included many more issues that crossed system boundaries in a multi-service repo.

In terms of one-shot "follow these detailed instructions" I think we are at a point where frontier models are overkill, especially at the price. For broad fan-out explorations, I think we are just getting started.

I was worried about some messup with taxes (my tax advisor messed up here), I managed to get it sorted on time - this was super naive but in -germany when they send you the taxes they mention the "cents" place as well in a very weird way, I assumed that was the entire number and assumed the tax issue I had was 10x larger than the amount it really was (10x and not 100x as my brain is in a place of pressure due to other circumstances and I wasn't thinking clearly)

GPT 5.6 incorrectly stated that I had nothing to do, Fable got the issue correctly and I was able to see that that was indeed the cents place and that I was more worried than I realized, and I managed to get a temporary solution setup (that I verified and I am sure is correct).

Which is to say it managed to relieve me of quite a bit of stress haha

https://squishy.franzai.com/ - fable 5

even with opus 4.8 it could not come up with the necessary algos to reliable create harder and harder levels - in the now famous big fable 5 gap of 2026 opus even screwed up the algo a few times and needed to recover it from git.

I have been experimenting with dissassembly and rewrite of 90s dos games in Rust.

It's not something I have experience in so I rely heavily on the models setting the direction and automating the work.

It's something where the level of intelligence has a huge impact. I still haven't succeeded, it's mostly an experiment for me to test the edges of model capabilities and see how they tackle such a hard problem.

I feel like the converse question is more relevant: what projects have open source agents been good enough to implement on their own?

I wanted to see how Fable could do at getting Linux running natively on my M4 Mac mini. Turns out pretty well: it patched the Asahi installer and m1n1 bootloader to support M4, and created a workflow where it could build, push, and debug kernels with no interaction on my part after the initial setup. It's now happily running headless with ssh access, and hopefully will have video soon after a USB DisplayLink adapter arrives. (No HDMI/Thunderbolt yet).

I don't know for a fact that earlier models wouldn't be able to do that, but I figure if they could we would have heard about it by now.

It all depends on how hands on you are willing to be.

If you are using the model to write to code faster with extensive human oversight you can develop a lot faster using the non-frontier models. I was doing that extensively last summer.

But now my thought process is I want to focus on architecture and product direction. I have not seen Sonnet level models be capable of performing autonomously enough to take a feature end to end reliably enough to be completely hands off. In fact there a many cases where Opus will fail as well where Fable will succeed.

Of course that is not to say that Fable will always do things correctly. It will happily take an under-specified problem statement and happily use up all of your usage to build the wrong thing, while Opus at least recently stops constantly to check in.

it seems we should be storing chains of prompts instead of code, and then replay them when a new model is released.

Coming up with good French grammar quizzes. GPT5.6 is noticeably better.

Older models usually pitch the difficulty too hard/easy and rely on me remembering the gender of words.

I’ve not noticed a big programming improvement with the newer models.

I haven't seriously used GLM/Qwen/etc for agentic work. I will say, that the GPT and Claude models from 6 months ago were very usable within their respective agentic harnesses (this matters a lot). But there is no way you could convince me to go back to GPT-5.0 from GPT-5.6 for a serious amount of agentic work.

I could probably accomplish most of the same tasks with GPT-5.0, but it would take a lot more involvement from me, more troubleshooting, and significantly more time.

I don't bother to use open-weight models, but for all of the "security" or "security adjacent" work I have tried recently, GPT 5.5 x-high has been the first model that produced useful output.

Even very old models could spot the most glaring issues, but it's a different story if you scan a source repository where humans can't find security vulnerabilities even after hours of reading through the code. Feed something like that to, say, Gemini Pro 3.1 and you'll get a bunch of false positives back, nit-picking, or variants of "this could be insecure if the code around it changes in unreasonable ways in the future".

Feed the same thing into GPT 5.5 x-high and then tens of minutes later it'll find half a dozen unauthenticated remote code execution vulnerabilities, arbitrary file read/write vulnerabilities, or similar.

Until it got nerfed, Mythos was similarly a huge step up for a lot of people working on code security.

6 months behind is probably a bit rich, I find the frontier models more capable in sustained agent and compiler tasks. Simple tool-chains and basic reasoning seems more or less on par across the board, at least for my use cases.