We Automated Everything Except Knowing What's Going On
45 points
1 hour ago
| 16 comments
| eversole.dev
| HN
bluGill
39 minutes ago
[-]
You cannot understand everything. That has been the case since long before AI. I have a vague idea how the linux kernel works, and I could figure it out (I once found and fixed a bug in FreeBSD device drivers) - but I don't, I just trust it works. I've never looked at sqlite to understand how it works - I know enough SQL to be dangerous and trust it works. I know very in depth how the logging framework of my project works - maintaining that code is part of my day job and so I need to know, but the hundreds of other developers in the company that use it trust it works. Meanwhile my co-workers are writing code that I don't understand, I trust they do it well until proven otherwise.

AI is very useful, but it so far doesn't write the type of code I can trust. Thus I use it but I carefully review everything it does.

reply
kennethops
35 minutes ago
[-]
>You cannot understand everything.

I 100% agree with this in a individual person sense, but in a humanity sense someone does understand linux very deeply and is very intentional on how they change it which to me is how I gain trust in it.

does trust change when the entire SLDC is AI?

reply
bluGill
9 minutes ago
[-]
Linux is less then 40 years old. Most of the people who designed it are still alive. How will the situation be in 40 years when the current maintainers are dead? (reiserfs comes to mind - it was just becoming great when [censored] and filesystems in linux went backward for many years, will that be allowed to happen next time?)

There are systems still in use from the 1960s (maybe before) - the original authors are at least retired and likely dead. I question how well the replacements understand all that. Sure they have had to dig in and understand some parts, but what about the parts that just keep working and don't need new features?

reply
afry1
1 hour ago
[-]
"The future belongs to whoever understands what they just shipped."

Perfect summary.

It's like we invented a world where you can finally, _finally_ speedrun an enormous legacy codebase and all patted ourselves on the back like that was a good thing.

reply
iammjm
38 minutes ago
[-]
We are speedrunning legacy "codebases" all the time. Or do you conjure up your own pickaxe, mine your own minerals, produce your own electricity, and construct your own computers and networks first before you go off to develop an application? Would you even know how to do those things? That is all enormous legacy codebase that we speedrun all the time. Just add one more to it.
reply
scared_together
5 minutes ago
[-]
That’s all legacy but none of it is speedrunning.

If we could conjure pickaxes and electric power plants in a single day, that would be speedrunning.

reply
allenu
43 minutes ago
[-]
Yup. AI can't automate long-term responsibility and ownership of a product. It can produce output quicker but somebody still has to be responsible to the customer using said product. The hard limit is still the willingness of the human producing the code to back what's been output.
reply
2OEH8eoCRo0
1 hour ago
[-]
I like that. These AIs are legacy codebase generators. Nobody knows how it works and everyone is afraid to touch it.
reply
silverquiet
1 hour ago
[-]
Anything in production is legacy; I'm pretty sure it happens as soon as the code is shipped regardless of who wrote it.
reply
alanbernstein
1 hour ago
[-]
True, but I think there's another dimension implied: how many devs are left that understand the code? Being able to start at zero is a fascinating surprise (compared to five years ago).
reply
kennethops
37 minutes ago
[-]
a comment I cannot stop thinking about is "we need to start thinking about production as throw away" Which is a wild thought to think about when I think about on my career. We had so many dbs or servers that we couldn't touch because they were a special snowflake
reply
andai
1 hour ago
[-]
There's a funny angle to all this. There was an article last year where the author asked AI for a web app. It installed a gigabyte of node modules and crashed on startup.

He told it to calm down and just use php, it gave him 100 lines with no dependencies that worked the first time.

The Pieter Levels stack :)

Of course, this is ideal for solo entrepreneur. If you are employed, then you cannot finish it in 100 lines. How will you get paid to maintain it for the next ten years, and hire all your friends to help you?

I think this difference in incentives explains most of what we've been complaining about for the last twenty years.

reply
iammjm
42 minutes ago
[-]
Should the goal really be to build a system that we completely understand, or build a system that solves a problem? Like we dont fully understand quantum physics, yet good enough to build helpful systems on top of it. Or like not knowing exactly what every bee in a hive does at any moment, yet still reliably harvesting honey in the end? I think people have this modernist desire for absolute truths and certainty, where the world we live in clearly is postmodern. There are no certainties, only probabilities. So embrace the chaos, try to build systems that help to contain entropy for some useful purposes, and accept that all of them will eventually fail in some way and you will need to course correct. Faulkner is dead, long live Pynchon
reply
bluGill
36 minutes ago
[-]
Someone needs to verify it works enough to trust the output. Of course some things are more critical than others. I don't worry too much about a badly written game, just is it fun - but I still don't want it to delete or transfer my money while I'm playing. However there are also systems where people die if it fails and those need to have a lot more trust/understanding.
reply
bluetomcat
1 hour ago
[-]
We went from expressing computation via formal, mostly non-ambiguous languages with strict grammar and semantics, to a fuzzy and flaky probabilistic system that tries to mimic code that was already written. What could go wrong?
reply
seethishat
28 minutes ago
[-]
Abstractions have been happening since the 1970s when ASM was replaced by the C Programming Language. From there we got C++ (look, it actually has a string type that most humans understand!) then we got memory safe managed languages like Go that is almost human readable, runs almost everywhere and doesn't have buffer overflows.

ASM was machine specific. C was portable but required expert programmers. C++ was even more user friendly, but still very hard for normal people. Today, most anyone can write a program in Go.

The more we abstract, the less knowledge/expertise is needed. So yes, programs are being built by people who don't really understand what they are doing. That is intended.

reply
whynotmaybe
35 minutes ago
[-]
I'm still balancing on whether we "need" to know what's happening.

Very few understand deeply what's happening within the computer between the cou and the bridges and the rest.

The fdiv bug in 1994 took us all by surprise because we were in a situation where bug couldn't exist in hardware, because it either works or it doesn't.

When I'm using firebase or aws, I don't know the underlying system, I don't know why some resources can be created with an underscore or other can't start with a number.

Yet it works.

We're working in layers where usually we only touch the last one. Yes, understanding the others is great to debug.

I'm even wondering whether we need tests when they are written by the same llm that wrote the code.

reply
bluefirebrand
29 minutes ago
[-]
> I'm still balancing on whether we "need" to know what's happening.

Of course we do. Otherwise we start trying to water crops with Brawndo

> Very few understand deeply what's happening within the computer between the cou and the bridges and the rest

But it's very very important that those people deeply understand it. We cannot replace their actual knowledge with LLM approximations of their knowledge

reply
andai
1 hour ago
[-]
The whole point of AI is that we don't have to think anymore. Knowing what's going on is the AI's job.

Not saying that's how it should be, that's just the world I predict in the not too distant future.

I love thinking, but most people I know seem to experience it as a form of physical pain.

reply
climike
1 hour ago
[-]
In a similar fashion it appears that article was automated - did the author read every word in their own article?
reply
kennethops
1 hour ago
[-]
I tried to write it more stream consciousness
reply
DaedalusII
1 hour ago
[-]
eventually i realised it was cheaper just to vibecode and buy put options over my company

by managing the risk of failure and technical debt with a financial instrument, i have a lot more freedom to move fast and break things, and scale aggressively

reply
dvsfish
56 minutes ago
[-]
Would this not be considered insider trading? Serious question
reply
DaedalusII
39 minutes ago
[-]
it is if you have specific event driven knowledge that the put options would gain from

but if you just buy deep out of the money put options every year, say 50% price drop, in a 10b5-1 structure, its ok. you will get sued a bit more than usual

not much different than buying life insurance, except if your company crashes like monday.com , some schmuck who sold you the puts will have to pay you a tonne of cash then you can do a dilutionary rights issue or just use the cash buy a boat in miami and start something else

reply
frisia
1 hour ago
[-]
Actually unreadable
reply
kennethops
1 hour ago
[-]
Im not a writer. Just a guy with an opinion
reply
digital-cygnet
54 minutes ago
[-]
I like the thought behind the piece, but what I think the criticisms are reacting to is the profusion of short, bursty sentences (just like the ones in the parent post), which can be great when used sparingly, but start to feel repetitive and have a "LinkedIn"-ish vibe, at least to me. For example the very end:

  Most of you won't be able to answer that. And you already know it.
  
  That's the conversation this industry needs to have. Not tomorrow. Now.
I hope you don't take this the wrong way and do continue writing - I enjoyed this piece, just wanted to give some constructive feedback
reply
kennethops
41 minutes ago
[-]
I have been doing most of my writing over there...honestly I hate it. So thank you for the feedback.
reply
Flashtoo
59 minutes ago
[-]
Then just post your opinions rather than the text the LLM dreamed around your opinions. Short posts and tweets tend to be well-liked on HN, there is no need to puff it up to a big blog post.
reply
frisia
58 minutes ago
[-]
Look, I'm sympathetic to not feeling like you're a good writer, but there are plenty of writing styles which doesn't turn your opinions into overly dramatic AI slop. And now I don't even know which opinions are your own and which are from a GPT, hence my "unreadable" comment even if it sounds harsch. But it literally is impossible to infer what your opinions actually are when they have been butchered this hard in slop.
reply
allenu
1 hour ago
[-]
It would be so much easier just to see the three or four bullet points given to the LLM than to read it.
reply
leecommamichael
40 minutes ago
[-]
Whoa a CEO writing about why their product is especially important in this very moment!
reply
Rygian
1 hour ago
[-]
The title reminds me of the single lesson I retained from a training for upcoming people managers: "You can delegate everything except accountability."
reply
gwynforthewyn
1 hour ago
[-]
Honestly, the post itself reads very generated, very rage bate. I have so much more faith in us and our hobby/industry than this blog post.

There are reports of industries trying to use these tools to generate as much as possible, sure. There are also people generating bad art and unpleasant prose and using llms to generate nonsense they don’t pay attention to.

I don’t see why that implies that you or I lost interest in tinkering with toys we build. If I want to spend 4 weeks understanding oauth a little better by implementing a client, I still can and I still do.

Automating our builds absolutely didn’t create a cathedral of complexity while nobody noticed. It did mean I can open an Free Software project, read the build file and understand how to build the thing. That’s the opposite of generating complexity.

I worry about our future generations as much as the next person, but this low effort pabulum doesn’t represent the thoughtful industry and hobby that I love.

reply
raesene9
50 minutes ago
[-]
I don't think the hobbyist interest will go away, but I can see what's happening affecting business that use software.

For most businesses, software is just a means to an end, they don't really care how high quality and thoughtful the systems they use are (e.g. look at any piece of "enterprise" software)

What LLMs have done is made much much easier for orgs to launch new features and services both internally and externally, without necessarily understanding the complexity.

For me, that's what this post tapped into. Many orgs already have more complexity than they can reasonably handle. Massively accelerating development, is not going to make that problem better :)

reply
kennethops
57 minutes ago
[-]
>Honestly, the post itself reads very generated, very rage bate. I have so much more faith in us and our hobby/industry than this blog post.

Don't get me wrong. I think the future is very bright for software. I have friends who are scientist and biomedical professionals and I am excited to see what they are able to do with the powers of software where they don't need to care about syntax and can only lead with their intentions.

The rage bait part is mainly my frustrations manifesting. As an SRE my annoyances came off a little bit when it comes to how fast developers are shipping vs how fast our rails can maintain things.

reply
bena
1 hour ago
[-]
I keep seeing the canard that "Anyone with an idea and access to an AI agent can ship a product. What used to take a team of twenty and six months now takes one person and a weekend. That's not hype. It's happening right now, everywhere, all at once."

But I don't see it. Where is this glut of software?

reply
ryanmcl
1 hour ago
[-]
I'm one of those people. Taught myself Rails 8 months ago at 45 with zero coding experience. I now have a production app with real Stripe payments, AI voice cloning, background audio processing, and a PWA with push notifications. Live users, live transactions.

Before AI assistance this simply wasn't in my possibility space. Not because I couldn't think through the problems, but because the gap between "I know what I want to build" and "I can actually build it" required years of skill acquisition I didn't have while raising my son and being a good husband (after some rough years ay).

The glut isn't visible yet because most of us aren't shipping to HN. We're shipping to our tiny audiences, our friends and family, our niche communities. The software exists ...it's just not venture-scale so nobody's writing about it.

reply
matltc
1 hour ago
[-]
It is not true. Speaking from experience: family member tried this and they couldn't get past a landing page.

I suppose if one is simultaneously ignorant when it comes to software and an expert at agentic workflow, then yeah sure, maybe--at the cost of how many tokens, though? But logically it seems that the former would preclude the latter.

Also, the "get it done in a weekend" seems to be a gross exaggeration.

reply
kennethops
1 hour ago
[-]
I am mainly seeing across a lot of my engineer friends and mentor who I respect deeply. They are using swarms of agents to build crms, small business and run their homelabs.
reply
caseyohara
1 hour ago
[-]
Where are these small businesses and startups? The software economy should be booming, right? I’m not seeing it.
reply
raesene9
46 minutes ago
[-]
There's a massive difference between launching a piece of software and launching a successful business.

Over the last couple of months I've seen a load of new "product launches" in my niche but when you look at them they're largely vibecoded and don't show deep understanding and sustainability, so it's pretty likely you'll never see them as successful businesses.

Looking at some of the related places like /r/sideproject/ there's a lot of releases and I'd be willing to suggest that most of them are using LLMs

reply
caseyohara
19 minutes ago
[-]
Then, respectfully, what is the point? Does the trillions-of-dollars AI industry exist to support a few hobbyists building niche products to scratch their own itch? I thought the promise here is increased productivity, presumably in the economic sense.

There seems to be a lot of hype, and has been for years, but I’m not seeing it materialize as actual economic output. Surely by now there should be lots of businesses springing up to capture all of this value created by vibecoded software.

reply
cowlby
59 minutes ago
[-]
Ever since Opus 4.6 came out, I've "vibecoded" a bunch of personal apps/CLIs that would've taken me months before. Some examples:

- CLI voice changer with cloned Overwatch voices on ElevenLabs.

- Brother P-Touch label maker using HTML/CSS. Their app is absolutely atrocious.

- Converted a FileMaker CRM into a Next.js/Supabase app.

- Dozens of drag-n-drop or 1-click/CLI tools. Think flattening a folder, a zip file.

- Dozens of Chrome Extensions and TamperMonkey user scripts. Think blocking ads with very targeted xpath.

But when I think about sharing them it feels like what's the point since anyone can make them themselves?

reply
Traubenfuchs
1 hour ago
[-]
This person has never worked with several decades old government, bank or tax (return) code where all that's ever done is edge cases and implementations of new laws and capabilities being forever bolted on to each other. Systems that were half migrated from a PL/I / Cobol mess to java 7 by Accenture, until the money ran out with the result being that both systems now exist forever and have to be integrated into each other for years. In the end you have decades old code bases maintained by people with less than 10 years of total work tenure, who will leave for greener pastures soon. No one to ask but some old grumpy grey beard with a royal salary who barely does any work but has some ancient wisdom to share.

No one understanding what's going on inside of complex systems in financially constrained environments built and maintained by average, at best, engineers is the norm and is what keeps the world running.

None of that is a symptom of AI. The only change AI brings is that even first person developers don't know anymore what the fuck they just deployed.

reply
kennethops
55 minutes ago
[-]
Cries in defense contractor nosies. (I have)

I wanted to touch on this point but then this post started getting WAY too long.

>the only change AI brings is that even first person developers don't know anymore what the fuck they just deployed.

This to me is going to make your first point 100x worse in every damn way

reply
simgt
43 minutes ago
[-]
I agree with all of this, but that's assuming we've reached a plateau. Maybe Claude 6.3 will be able to churn through 10M lines of Java and Cobol, tidy it and convert it to Rust. Or maybe not, but so far the scaling laws are holding.
reply
terseus
53 minutes ago
[-]
AI may not be the source of the problem but can make it a hundred times worse.
reply