I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders and worked on the first try.
Afterward, I sort of had to reflect on the fact that I learned essentially nothing about building vector search. I wanted the feature more than I wanted to know how to build the feature. It kept me learning the thing I cared about rather than doing a side quest.
You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works.
My hunch is that the 30 minutes of focused learning spent with a custom-built version that solves your exact problem is as effective (or even more effective) than four hours spent mostly struggling to get something up and running and going down various rabbit holes of unrelated problem-solving.
Especially if realistically you were never going to carve out those four hours anyway.
Of course, this kind of interactive deep engagement with a topic is fast becoming obsolete. But the essence to me of “knowing” is about doing and experiencing things, updating my bayesian priors dialectically (to put it fancily)
I don't think that's incompatible with getting help from LLMs. I find that LLMs let me try so much more stuff, and at such a faster rate, that my learning pace has accelerated in a material way.
Something I'm really interested right now is the balance in terms of the struggle required to learn something.
I firmly believe that there are things where the only way to learn how to do them is to go through the struggle. Writing essays for example - I don't think you can shortcut learning to write well by having an LLM do that for you, even though actually learning to write is a painful and tiresome progress.
But programming... I've seen so many people who quit learning to program because the struggle was too much. Those first six months of struggling with missing semicolons are absolutely miserable!
I've spoken to a ton of people over the past year who always wanted to learn to program but never managed to carve out that miserable six months... and now they're building software, because LLMs have shaved down that learning curve.
But when it replaces you struggling through figuring out the mental model of what you're doing, then I think you end up learning at a much more shallow level than you would by doing thing manually.
Is it a substitute for actually learning how to look at objects and break them down into shapes and color and value? No. You gotta put in the work if you want the result. Brains just work like that.
"Struggling with semicolons" isn't any different than drawing a hundred derpy looking faces that look terrible.
Ira Glass has a quote about this:
> “Nobody tells this to people who are beginners, I wish someone told me. All of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it’s just not that good. It’s trying to be good, it has potential, but it’s not. But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you. A lot of people never get past this phase, they quit. Most people I know who do interesting, creative work went through years of this. We know our work doesn’t have this special thing that we want it to have. We all go through this. And if you are just starting out or you are still in this phase, you gotta know its normal and the most important thing you can do is do a lot of work. Put yourself on a deadline so that every week you will finish one story. It is only by going through a volume of work that you will close that gap, and your work will be as good as your ambitions. And I took longer to figure out how to do this than anyone I’ve ever met. It’s gonna take awhile. It’s normal to take awhile. You’ve just gotta fight your way through.”
If companies decide that professional learning is unnecessary in the age of AI they'll be committing a horrible blunder. Their "fuck around" phase might sting, but missing an entire generation of skilled professionals is going to make our value skyrocket in the "find out" phase, a few years down the line.
I still think paint by numbers is a valid early step along the path to learning to draw.
I think the whole discussion about coding agent reliability is missing the elephant in the room - it is not vibe coding, but vibe testing. That is when you run the code a few times and say LGTM - the best recipe to shoot yourself in the foot no matter if code was hand written or made with AI. Just put the screw on the agent, let it handle a heavy test harness.
New skills, not no skills.
There will still be a wide spectrum of people that actually understand the stack - and don’t - and no matter how much easier or harder the tools get, those people aren’t going anywhere.
You can't really go to giant piles of technical debt and look to those for places to be human. It's soul-destroying. My concern would be that vibe coding will make those places of soul-less technical debt even deeper and deadlier. There will be nobody there, for generations of cruft. Where once the technical debt was made by committee, now it'll be the ghosts of committees, stirred up by random temperature, only to surface bits of rot that just sink down into the morass again, unfixed.
When 'finicky' is actually an interesting problem, or a challenge, that's one thing. When 'finicky' is just 'twelve committees re-hacked this and then it's been maintained by LLMs for years', there is nothing gained by trying to be human at it.
Ultimately it comes to whether gaining the know how through experience is worth it or not.
At some point things will get hard, as long as the world is. You don't need to concern yourself with any technical layer for that to be true. The less we have to concern ourselves with technicalities, the further that points shifts towards the thing we actually care about.
As an apprentice, you get correct and precise enough instructions and you learn from the masters perfection point downwards.
Maybe we have reached a point where we can be the machine's apprentices in some ways.
I learned refactoring patterns from Fowler's book. But when I tried to actually use them I still struggled. I didn't fully understand how the patterns worked until I actually tried (and failed) to use them a few times.
You don't really internalize things until you understand what doesn't work just as much as what does. You don't learn nearly as much from success as you do from failure. I would say the ratio of truly internalized knowledge is much higher for failure.
The notion that you can get a bot to just vomit out a vector database and then you can just "read the code" and you'll understand how a vector database works is just ludicrous.
I could change the post to be about learning word-working by watching a robot build a shelf and nothing would change.
Turns out there's science that backs me up here: https://en.wikipedia.org/wiki/Worked-example_effect - showing people "worked examples" can be more effective than making them solve the problem themselves.
That Wikipedia article is a little weak, this MIT page is better: https://tll.mit.edu/teaching-resources/how-people-learn/work...
That’s not what having a bot generate your integration is and reading it post-facto is. The bot isn’t guiding you through the process so you can go do it yourself. At best you would use this as a reference to go do another integration yourself - but at this point why even bother when you can just get the bot to do it again?
The only thing people learn using AI is how to do things with AI.
It is if you ask it to. Learning well with LLMs requires a lot of self-discipline - you have to be actively aware of the threat that you won't actually learn effectively and take steps to counter that.
I keep meticulous notes of everything these things do for me, which adds up to a valuable set of notes over time. I gave up on remembering things without notes a long time ago!
I can only speak for myself, but the only way I've been able to learn things rapidly in this industry is by writing things myself: even rote re-typing of books or SO answers was enough to trigger this for me.
Just querying models and reading output doesn't seem to work for me, but that's maybe down to my particular learning style.
I now know what the model does (at a black box level) and how all the parts fit together. And I have plans to build classifiers on top of the vectors I built for further processing.
The experience of fighting Python dependencies gives me more appreciation for uv over venv and will leave me less stuck whenever the LLM fails to help resolve the situation.
LLMs can hurt less experienced engineers by keeping them from building an intuition for why things work a certain way, or why an alternative won't work (or conversely, why an unconventional approach might not only be possible, but very useful and valuable!).
I think problem solving is optimization in the face of constraints. Generally using LLMs IME, the more you're able to articulate and understand your constraints, and prescriptively guide the LLM towards something it's capable of doing, the more effective they are and the more maintainable their output is for you. So it really helps to know when to break the rules or to create/do something unconventional.
Another way to put it is that LLMs have commodified conventional software so learning when to break or challenge convention is going to be where most of the valuable work is going forward. And I think it's hard to actually do that unless you get into the weeds and battle/try things because you don't understand why they won't work. Sometimes they do
What I don't believe is that it HAS to be like this. Maybe it's my natural optimism showing through here, but I'm confident it's possible to accelerate rather than slow down your learning progress with LLMs, if you're thoughtful about how you apply them.
An open question for me is how feasible it is to teach people how to teach themselves effectively using this new technology.
I have a core belief that everything is learnable, if people are motivated to learn. I have no idea how to help instill that motivation in people who don't yet have it though!
It's not really an open question. We've had a huge amount of content on the internet including documentation, tutorials, example code, and actual online courses available for years and in the end most people don't learn effectively when presented with that information and left to themselves. LLMs are no different.
Stifle the tools, somehow?
You’ve had nontechnical devs since npm, or before!
No: people that care to understand the whole stack, and be able to provide that value, will still exist and shine.
My hunch is the exact opposite of this. You will learn close to nothing by reading this for 30 minutes.
I find it hard to believe that wasting hours hunting for a missing semicolon (at the very real risk of quitting entirely) is essential for learning. Does that mean every student who asks a TA or fellow-student to help them find that semicolon is hurting themselves when they do that?
If not, what's different about asking an LLM?
The most relevant concepts appear to be:
- Desirable Difficulties - https://en.wikipedia.org/wiki/Desirable_difficulty - "A desirable difficulty is a learning task that requires a considerable but desirable amount of effort, thereby improving long-term performance. [...] The task must be able to be accomplished. Too difficult a task may dissuade the learner and prevent full processing."
- Worked-example effect - https://en.wikipedia.org/wiki/Worked-example_effect - "Specifically, it refers to improved learning observed when worked examples are used as part of instruction, compared to other instructional techniques such as problem-solving. [...] However, it is important to note that studying [worked examples] loses its effectiveness with increasing expertise"
- Expertise reversal effect - https://en.wikipedia.org/wiki/Expertise_reversal_effect - "The expertise reversal effect refers to the reversal of the effectiveness of instructional techniques on learners with differing levels of prior knowledge."
- "Generation effect" - https://en.wikipedia.org/wiki/Generation_effect - "The generation effect is a phenomenon whereby information is better remembered if it is generated from one's own mind rather than simply read."
Engaging with an intellectual problem, trying to solve it one way, failing, reasoning through the process and the requirements, trying to discover a better way of solving something, going down some wrong paths, backtracking, merging diverging ideas and ultimately finding a solution is going to yield an infinitely deeper understanding of the problem, what works, what doesn't, and improve your general intuition and problem-solving skills.
Deep engagement builds deep understanding, shallow engagement builds shallow understanding. There's no substitute for doing the hard work yourself - I've tutored classmates in school and I find this rather obvious. A tutor (human or LLM) can try to find a way to explain something in a way that you understand but if you don't do most of the hard work yourself it's never going to stick. I noticed that when I would spoon-feed answers to people it would always just lead them into a false sense of confidence.
There's a reason kids do better when assigned a 1-1 tutor. LLMs, used effectively, can have a similar effect. Probably a weaker effect although maybe it can be stronger since there's no shame involved in asking an LLM a question.
Try reading tutorials on a new programming language for 30 minutes and then open new text file and write basic loop with print.
It won’t even compile- which shows you haven’t really learned anything. Just read an interesting story. Sure you pita few bits here and there but you still don’t know how to do even the moat basic thing.
It's more like having the tutorial author there with you and actively engaging with them to collaborate on building the exact tutorial for the exact project you're looking to build.
I'll take a bidirectional conversation with a subject matter expert (and when you're just staring to learn Rust LLMs can absolutely take the role of "expert", in comparison to you at least) over struggling on my own against static documentation and the Rust compiler.
And I can take over the wheel at any moment! It's entirely on me to decide how much I get to do vs how much the LLM does for me.
That's why learning in this way is a skill in its own right, and one that I'd like to see studied and formalized and taught to other people.
I have a hard time imagining how much you'd have to literally bribe me to get me to try doing it the way you describe. I'm too interested in implementation details of things and looking for innovations—in fact I make my living doing that, like some cyberpunk gremlin just delighting in messing with stuff in unexpected ways. I don't understand why you're not, but maybe it's not for me to understand.
Carry on. We'll check back and see how it worked for ya :)
I ported a complex new Python program I wrote to Go last night on my phone out of nothing more than wild curiosity. You can bet I learned a bunch about Go in the process, from bed, in about 20 minutes.
Great for Opus because you’re now a captive customer.
you might be a little too far removed from the situation to comment authoritatively?
I have a side project that I started in January 2024. Initially, used Github Copilot autocompletions heavily. This year I started using CLI agents (mostly Claude, but others too) to do more stuff. I got to around 100k LoC (sure, it's not enterprise scale, but for a personal project it's pretty big), but I'd argue it's maintainable, it's split into 10 Django apps, that are each pretty self contained, I've done several refactors on it (using AI agents) to make it more maintainable.
I think that could work, but it can work in the same way that plenty of big companies have codebases that are a giant ball of mud and yet they somehow manage to stay in business and occasionally ship a new feature.
Meanwhile their rivals with well constructed codebases who can promptly ship features that work are able to run rings around them.
I expect that we'll learn over time that LLM-managed big ball of mud codebases are less valuable than LLM-managed high quality well architected long-term maintained codebases.
Honestly, i’m making stuff up, as I don’t think it’s feasible right now because of the context sizes. But given how fast things develop, maybe in a couple of years things might change.
We recently started working on https://github.com/accretional/collector to serve as a dynamic proto ORM+CRUD server with search and discovery, and features for operating as a node in an "agent/service mesh". The idea is that you can create a uniform interface for data retrieval/search/APIs that lets agents dynamically register, invoke, or discover any data type or service, or write it themselves, then register it locally or share it.
It is feasible to do this stuff now actually, just a bit tricky because most LLMs aren't trained to operate this way without very explicit instructions for how to do so, and for collector specifically the API surface is probably too big. But I am pretty sure neither would take long to fix if enough people were adopting this kind of pattern.
But yeah, what you guys are doing looks sweet! I need to get out of my ass and see what people are doing in this sphere as it sounds fun.
I'd love to do that too but there are basically three ways to teach LLMs how to use it afaik: with data created "in the wild" and a degree of curation or augmentation, or with full-on reinforcement learning/goal-oriented training, or some kind of hybrid based on eg conformance testing and validating LLM output at a less sophisticated level (eg if it tries to call an api that's not in the set that it just saw during discovery, the LLM is being dumb, train it out of doing that).
The thing is they are not really mutually exclusive, and LLM companies will do it anyway to make their models useful if enough people are using this or want to use it. This is what's happened already with eg MCP and skills and many programming languages. Anyway, if prompting works to get it to use it properly it validates that the model can be trained to follow that process too, the same way it knows how to work with React
vs
“we’ll figure it out when we get there” human slop.
But I’d say some projects (I expect to live less than 1 year) I’d just vibe code them so I won’t care much about the code. I just give very high level architectural ideas and that’s it.
Other projects which I expect lifespan to be more than 1-2 years I won’t let it become a ball of shit.
So it depends on the project.
I get that even now it's very easy to let stuff get out of hand if you aren't paying close attention yourself to the actual code, so people assume that it's some fundamental limitation of all LLMs. But it's not, much like 6 fingered hands was just a temporary state, not anything deep or necessary that was enforced by the diffusion architecture.
The reason software is so valuable is that it's capital/up-front investment in figuring something out that can continuously deliver value with low or no marginal cost. Rewriting/maintenance/difficulty/figuring out software is marginal cost.
AI is still incredibly useful used in tandem, but have it implement full feature from one sentence usually lead to doom.
The other day I generated an MCP server for AST of Java. I had no clue how that works. I couldn’t test it because I had no idea how that looks like. Btw, AI even lied in tests, because it literally mocked out everything from live code. So everything was green, and literally nothing was tested, and it was untestable manually by me.
Opus/Anthropic is hands down the best in my experience. But using it feels like intellectual fast food (they all are), I hate the fact that I can build something like a neatly presentable one off spa tool (ty Simon) when I'm barely paying attention. it feels unsatisfying to use.
EDIT: because I'm rambling, I like "AI" as much as the next guy, probably more because I was there before it turned into LLMs"R"US, but I also like(d) the practice of sitting around listening to music solving problems with Scala. I don't know why we've decided to make work less fun..
There are just too many parts involved to do anything. For example today I built a simple data collection app to use on my phone that involves inventories with photos for a tedious workflow I have to do. I knew what I wanted but didn't know how to even choose which tools to bother learn. And just even trying things to see if an approach works or not without spending hours learning one thing or another or wading through the hell of web search is really great.
Things I learned today that I figure everyone else must know: if you want to take a photo from a webapp I guess you need https. So I decided to try mTLS (knew it existed but never had the time) so asked Claude to write me a short tutorial about setting it up, creating keys, importing them (including a cool single line trick of spinning up a python server and downloading the keys on my phone rather than find a USB stick or whatever). And then helping me figure out a path out of the suffering of Chrome and Firefox hating self-signed CA. But at least I figured out how to make Firefox happy. But it would insist on prompting me for the certificate for every htmx request. But chatting with Claude I learn caddy is pretty cool, it's go. Claude suggests an auth boxcar when I balk at adding auth and user management to my app because I think the webserver should handle all this shit (wtf is a boxcar? Claude clues me in). I tell Claude to use go or rust to build the boxcar because Jesus Christ "yay" build another service just to get a good damn customized CRUD app on my phone that can take a picture. Claude picks go which is fine by me. (Incidentally I can't write go, but I can read it and it's on my "to be learned" agenda and go seems safer than a pile of python for this simple thing) The boxcar was fine but Claude was struggling with getting headers to work in the caddy config. So while Claude is working on that I do a quick Google about whether caddy can have extensions because there has to be a better way to "if someone has authenticated successfully, give them a cookie that will last an hour so they don't have to mash the confirm about using the certificate for every goddamn htmx request" than spin up a web service. Interrupt Claude and suggest an extension instead of a boxcar. Claude's on board so we ditch the boxcar. Have Claude and codex evaluate the extension for security. They find important issues about things a jerk might do, fix them. So successful mTLS connections transition to session cookies. So my dumb CRUD tool doesn't have to worry about auth. Which it didn't have to do anyway except browsers say so etc because my phone is literally only able to access the server via VPN anyway.
Other things I have learned today that only wasted 5min of Claude's time rather than hours of mine: Firefox camera access can't control flash, focus or zoom. So call out to the native app instead.
This is all quite fun and the tool I'm building is going to really make my own life better.
Is there a better way to do this: probably.
I mean will you (we) retain all that it did after a few months go by? You may say we don't need to, but that sounds a little shallow given we're both on HN. Do you remember Gatsby's criticism of "Summer People"?
I'm pretty sure I will remember how easy and correct it was to modify Caddy vs the months of putzing around building Rube Goldberg constellations of services crap that I did last year for a different thing and that even Claude wanted to do. I've done the whole wading through outdated blog posts and trying to read documentation on other projects that I was doing. Learning five different projects and having to maintain seven services running in docker just so that I can use a tool to capture photos and store them in a webapp that can only be used if connected to my own VPN is insane and it's why I am not a web developer. I will 100% remember what 200 lines of golang does after looking at it again. The 1000 lines of JavaScript that were and backend auth crap that no longer exists: good riddance.
And no I don't trust my memory about what Gatsby said about Summer people without looking it up. I read Gatsby 30 years ago.
This is exactly what LLMs are great for. For instance, I'm looking at trading models. I want to think about buying and selling. I need some charts to look at, but I'm not a chart wizard. I can make basic charts, but it feels tedious to actually learn the model of how the charting software works. LLM will just give me the chart code for the visualization I want, and if I ever care to learn about it, I have it in a form that is relevant to me, not the form of the API documents.
In general, a lot of coding is like this. You have some end goal in mind, but there's a bunch of little things that need to be knitted together, and the knitting used to take a lot of time.
I like to say the LLM has reduced my toil while getting me to the same place. I can even do multiple projects at once, only really applying myself where there is a decision to be made, and it's all possible because I'm not sorting out the minutiae of some incidental API.
At best it plods along as you keep badgering Claude to fix it, until inevitably Claude reaches a point where it can't help. At which time you'll be forced to spend at least the 4 hours you would have originally spent trying to understand it so you can fix it yourself.
At worst the thing will actively break other things you do understand in ways you don't understand, and you'll have to spend at least 4 hours cleaning up the mess.
Either way it's not clear you've saved any time at all.
My proto+sqlite+mesh project recently hit the point where it's too big for Claude to maintain a consistent "mental model" of how eg search and the db schemas are supposed to be structured, kept taking hacky workarounds by going directly to a db at the storage layer instead of the API layer, etc. so I hit an insane amount of churn trying to get it to implement some of the features needed to get it production ready.
Here's the whackamole/insanity documented in git commit history: https://github.com/accretional/collector/compare/main...feat...
But now I know some new tricks and intuition for avoiding this situation going forward. Because I do understand the mental model behind what this is supposed to look like at its core, and I need to maintain some kind of human-friendly guard rails, I'm adding integration tests in a different repo and a README/project "constitution" that claude can't change but is accountable for maintaining, and configuring it to keep them in context while working on my project.
Kind of a microcosm of startups' reluctance to institute employee handbook/kpis/PRDs followed by resignation that they might truly be useful coordination tools.
My fun lesson this week was there's not a snowballs chance in hell GitHub Copilot can correctly update a Postman collection. I only realised there was a Postman MCP server after battling through that ordeal and eventually making all the tedious edits myself.
At the end of the day, you've spent just as much time gaining the knowledge, but one way was inductive (building it from scratch) while the other is deductive (letting the AI build it and then tearing it apart). Is one better than the other? I don't know. But I don't think one saves more time than the other. The only way to save time is to allow the thing to work without any understanding of what it does.
b) you also don't have enough information to say if it's saved you time because the costs you will bear are in the future. Systems require maintenance, that's a fact you can't get rid of with AI. And often times, maintaining systems require more work than building them in the first place. Maintaining systems tends to require a deep understanding of how they work and the tradeoffs that were decided when they were built.
But you didn't build the thing, you didn't even design it as you left that up to Claude. That makes the AI the only thing on the planet that understands the system, but we know actually the AI doesn't understand anything at all. So no one understands the system you built, including the AI you used. And you expect that this whole process will have saved you time, while you play games?
I just don't see it working out that way, sorry. The artifact the AI spit out will eventually demand you pay the cost in time to understand it, or you will incur future costs for not understanding it as it fails to act as you expect. You'll pay either way in the end.
Perhaps not. If LLMs keep getting better, more competent models can help him stay on top of it lol.
It's pretty amazing to watch tools-in-a-loop crunch away for >4 hours to solve a generally difficult problem through sheer brute-force.
This is of course quite highly correlated with an AI system being able to churn through a task for a long time. But it's not necessarily the same thing.
Of course the big questions are going to arise if/when we start passing lines like 8 hours (a whole work day) or 40 hours (a whole work week).
It's interesting because like the article notes, AI is really smashing benchmarks, but actual usefulness in automation of thought work is proving much more elusive. I think that collective experience of AI just not being that useful, or as useful as benchmarks suggest it should be, is captured in this metric.
"Human hours equivalent" itself is an interesting metric, because: which human? Or rather, I'm sure they had a coherent definition in mind: presumably a human reasonably competent at whatever the specific task is. But hours the abstract human standard would spend is different from the hours any specific person, say you or I, would spend.
In particular, some of the appeal (and risk!!) of these things is precisely that you can ask for help with things that would be quick work for someone (who knows jq, or a certain corner of the PyPI library ecosystem, or modern CSS, or TypeScript annotations, or something else) but not for you.
The second graph has this under it:
The length of tasks (measured by how long they take human professionals) that generalist frontier model agents can complete autonomously with 50% reliability has been doubling approximately every 7 months for the last 6 years...
I use Claude code and I get hit with a permissions prompt every 2 seconds for anything I try to do.
Sure I can turn off all dangerous permissions but it'd probably honestly stop and claim it's finished well before it actually is in most cases from my experience.
To be fair I haven't tried codex so maybe it's better at this but I'm my experience almost every model stops at some point and claims victory or stops and tells me something like "next we'll continue on with XYZ" at which point I have to prompt it to continue.
I have to specifically tell it to request permission for the command it wants to run, and then it works. Very annoying, and very annoying that it can't persist the permission, like Claude Code can, so it doesn't have to ask again every single time.
Thankfully the cloud versions (Claude Code for web, Codex Cloud) run like that already, and are relatively safe in that if anything goes wrong it happens on someone else's computer.
Perhaps the agent can find these themselves if told to make the code perfect and not just pass tests
I didn't include the TypeScript bit though - it didn't use TypeScript because I don't like adding a build step to my JavaScript projects if I can possible avoid it. The agent would happily have used TypeScript if I had let it.
I don't like that openElements = open_elements pattern either - it did that because I asked it for a port of a Python library and it decided to support the naming conventions for both Python and JavaScript at once. I told it to remove all of those.
I had it run a micro benchmark too against the before and after - here's the code it used for that: https://github.com/simonw/justjshtml/blob/a9dbe2d7c79522a76f...
BEFORE benchmark:
Input: 87,707 bytes
Average: 7.846 ms
Ops/sec: 127.5
AFTER:
Average: 7.769ms
Ops/sec: 128.7 (1% improvement)
> The current implementation uses a switch statement in step(). JavaScript doesn’t have proper tail call optimization (only Safari implements it), so true tail recursion would cause stack overflow on large documents.
I cannot imagine GPT5.2 working on a task for more than 2 minutes, let alone 4 hours. I’m curious if you’ve run into this and figured out a way around it?
I use prompts like this:
Build a pure JavaScript library (no dependencies) for encoding and
decoding this binary format. Start by looking at how the lite3-python
library works - the JavaScript one should have the same API and probably the
same code design too. Build the JS one in lite3-javascript - it should be a
single JavaScript module which works in both Node.js and in the browser.
There should be a test script that runs with Node.js which runs against the
files in the lite3-python/format_suite folder. Write the test script first,
run it and watch it fail, then build the JavaScript library and keep running
the tests until they pass.1. That’s the part of development work I hate the most and never really clicked with me
2. AI to to this point seems to be better at writing tests than code
Take this with the grain of salt that:
1. I suck
2. My work is mostly in the realm of infrastructure where testing has always been weird and a little dumb
But even if it wrote great tests every time, the trouble is that testing was designed around the idea of "double entry accounting". Even great tests can test the wrong thing. In the old world you would write a test case and then implement something to satisfy the same. If both sides of the ledger agree, so to speak, you can be pretty confident that both are correct. — In other words, going through the process of implementation gives an opportunity to make sure the test you wrote isn't ill-conceived or broken itself. If you only write the tests, or only write the implementation, or write none of it, there is no point at which you can validate your work.
If you have already built up an application and are reusing its test suite to reimplement the software in another language, like above, that is one thing, but in greenfield work it remains an outstanding problem of how to validate the work when you start to involve AI agents. Another article posted here recently suggests that we can go back to manual testing to validate the work... But that seems like a non-solution.
Others have pointed out your interpretation of long task is not the same as the article.
Maybe this is the negative effects of excessive LLM usage that are spoken about.
> I upvoted them for pointing that out.
I'm also curious about what you think about the GPs question. TBH, responding after reading half an article was a common thing for most people pre-LLM anyway.
If I had piped the article through an LLM first, I wouldn't have made the embarrassing mistake in that comment!
I read
It's "unlimited"
It's unlimited
What's interesting is the 50% vs 80% reliability gap. At 50% success rate on a 4-hour task, you're essentially gambling. If it fails, you've potentially wasted the 4 hours plus the time debugging why it failed.
This is why I think the current "agent" paradigm needs human checkpoints at regular intervals. Let the AI work for 30 minutes, then review progress. Repeat. This way you catch drift early before it compounds.
The other thing missing from these benchmarks: recovery ability. When the AI gets stuck on hour 3 of a 4-hour task, can it recognize the problem and backtrack? Or does it confidently continue down the wrong path?
At 50/50 it’s an ok bet if the debugging time is much less than the total human time, even if the loops are long, you might rather 4 hours of deep work on an important human thing or on just relaxing vs babysitting the LLM. Assuming that about half the time that will pay off with a correctly done thing with very little effort, it’s kind of amazing.
> What's interesting is the 50% vs 80% reliability gap. At 50% success rate on a 4-hour task, you're essentially gambling. If it fails, you've potentially wasted the 4 hours plus the time debugging why it failed.
Your first two paragraphs are at odds with each other. If it fails, you've potentially wasted the time it took the agent to *perform* the "it takes humans 4h" long task. Which in most cases is single digit minutes.
That's why one of the solid use cases for agents is doing multiple throw away proof of concepts to explore a problem / new feature before deciding on a solution to actually implement. Usually you'd have time for one, or maybe none. If it fails you've lost a maybe 10 minutes, but likely learned something new about the potential solution.
Solutions are quite easy to verify with differential testing and produce a number for direct comparison.
Less code is usually better and you generally can't "cheat" by adding more cruft so it nullifies the additive bias. Good optimization requires significant understanding of the underlying structures. Everything has performance tradeoffs so it requires systemic thinking and not just stringing independent pieces together.
So far I've found that Gemini Pro 3 was the best at reasoning about tricky SIMD code but the results with most models were pretty underwhelming.
Anyhow, I noticed more of a difference trying Opus 4.5 compared to Sonnet 4.5 than I'd noticed from, for example, the last couple Sonnet bumps. Objectively, at 1.66x Sonnet's price instead of the old 5x, it's much more often practical to consider reaching for than past Opus models. Anthropic's basic monthly thing also covers a fair amount of futzing with it in CC.
At the other extreme, another surprise of this family is that Haiku 4.5 with reasoning on is usable: better than Sonnet with thinking off according to some bencharks, and in any case subjectively decent for point edits, single-page thingies, and small tools.
The Conclusions section is not for making a sales pitch for your article. It is for summarizing any new knowledge the article brings out.
Extrapolating any exponential growth is always dangerous, but over say 3 years at this pace, we'd go from 2 hours to 70,or about 8 days' work.
Quite scary. But what does cost do over the same timeline? Does it increase with computational complexity? Is it worse - because, IIRC, transformers computational cost is quadratic in context length. Is it better - some kind of economies of scale?
I glanced thought the article but couldn't find any info on this.
It matches the my personal feeling when using progressively better models over time.
An infinitely unscrupulous model provider could double this five hour result by cutting your output tokens/second in half!
This isn't only a question of gaming the metric: the very strong current small-fast models (4.5 Haiku, Gemini 3 Flash) have no hope of being measured fairly against this - they will succeed or fail much faster just because they are much faster.
How about something like total output token count as the "long term horizon" metric instead?
The contrary is easily verifiable by everyone individually. It's nowhere near 100%, or even 50% for few minutes tasks even with the best models in real world situations.
So some model will misclassify my American black nightshade* weeds as a tomato, but I get consistently OK results for text out from good models unless it's a trick question.
* I recon, at least; looked like this to me: https://en.wikipedia.org/wiki/Solanum_americanum#/media/File...
What examples can you give of "real world situations" where they fail?
Obviously I don't want to use them for whatever that is.
And rarely is a software one and done, with a few round like this, the software architecture would have become schizophrenic. Combating this tendency usually require a lot of the work of these "long task" to be thrown away and more closely limiting what the AI is trying to do as they happen. The success of one "long task" is not necessarily a good thing!
I don't think I have 50% success rate at month long tasks.
Anything that exceeds one day is pretty hard.
METR currently simply runs out of tasks at 10-20h, and as a result you have a small N and lots of uncertainty there. (They fit a logistic to the discrete 0/1 results to get the thresholds you see in the graph.) They need new tasks, then we'll know better.
If true, how much of this is a result of:
1. Genuine technical advancement
or:
2. Shoveling trillions of dollars into compute resources in order to service incoming LLM requests in a way that is completely unrealistic over the long term?
In other words… are we talking about genuine, sustainable innovation that we get to take with us moving forward and benefit from? Or are we talking about an “improvement” that is more akin to a mirage that will eventually disappear when the Ponzi scheme eventually collapses?
The names Haiku, Sonnet, and Opus have not been chosen randomly.
.jpg){kind=link}