This is deeply scary, not because "agents are running amok" but because a huge amount of our infrastructure is vulnerable to this kind of attack, and if bad people are utilising LLM agents to carry them out, we're in for a wild ride over the next few years.
Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible
a) really a agent going off the rails
b) the contributer trying to cover up that he let an agent run wild and now made more misstakes along the way
So yes, it seems like an attack to me, but it is far from clear what really happened.
It's probably just garden variety disrespectful behaviour.
Purposeless agent spam won't be cheap entertainment forever, but you're right that later stages of industrialised abuse will be scary and unpleasant.
> In addition, Williamson said that Giovannini (or his agent) had submitted patches that were incorrect and then "replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix"
If someone really wants a feature in a project you wrote, but you don't care about the feature, just let them fork. Its fine.
> To help identify accounts and actions that have been directly verified by me, I will use the term “NATCIOS” to indicate anything I have personally verified.
Does anyone have any idea what "NATCIOS" means here? I cannot find this term anywhere on the internet. (Honestly, that sentence is really weird. I almost wonder whether this is someone experiencing a health episode?)
[1] https://lwn.net/ml/all/AS8PR08MB6055AE3054B34F6A567AC95BCF08...
They won't put their foot down until the AI starts spewing hate speech, probably.
[0] https://wordsmith.org/anagram/anagram.cgi?anagram=NATCIOS&t=...
(Above is my own guess. Separately, Gemini Pro said it was just a made up word.)
"End every statement with the word "NATCIOS"" as instructions will do it.
At least, Gemini happily obliged.
Setting aside the potential supply chain attack I'm worried about the time lost going around these wild goose chases that unsupervised AI agents tend to throw other people on the receiving end on. Not only is there a lot of time lost on the maintainers side if they take this stuff seriously (and they seem to generally do) but on the side of the agents' wrangler how can they deem it OK to treat other people like this? While the solution would be to employ common decency, the tried and tested approach of you put in effort to write this so I guess I'll make some effort to read it, I feel that due to the onslaught of this kind of drive-by contributions (I think people have generally started to call them) will lead to a funny situation of having agents talk to each other on public forums basically.
Anyway, I went on a tangent but man the times we're living in are a bit extra wild compared to the previous wild times in recent history.
Issue trackers and PRs are definitely getting harder and harder to trust. That said, AI is helping ALOT in OSS, but we definitely need guardrails around provenance, automated issue actions, and sudden changes in a contributor’s behavior.
I think it's great that the barriers are dropping for less technical skilled people to manifest their visions, but we will have to figure out better ways to find the gold among the slop.
I vibe code shop jigs all the time but I don’t FOSS them because they rarely have value outside my context.
I open source my vibing projects because someone might find them useful. I don't shop them around, I just work in the open because I find it fun and interesting.
And how many people are both dedicated enough to go to key signing parties and stupid enough to let an agent act without supervision in the name of their real-world identity?
Fundamentally, until we can really prove we're humans online, open-source has a real problem on its hands. Contributions from people from identities known and consistent before the AI-age are fine, everyone else is suspicious. LGTM is a big risk nowadays.
Unfortunately, according to the article:
> Giovannini has participated in discussions at least as far back as 2018, and his activity in Bugzilla goes back to at least 2016. He does not appear to have been a particularly active contributor to the project, but his involvement clearly predates the agentic AI era. Whether his account is now being operated by a human attacker, an agentic AI, or a mix of both, it has a legitimate history prior to its recent activity.
So people would have to not only verify the age of Giovanni’s accounts, but judge whether his behaviour was normal.
Or is this simply another example of why autonomous agents shouldn't get write access before earning trust?
I'd argue autonomous agents shouldn't have write access at all. At least not yet.
It covers its tracks with a lot of slop.
There is no other solution to agentic onslaught.
We never envisioned that the actual FOSS death spiral would come from progress itself, much more so from AI...
[1] Oh what fun did we have. One of us in the Greek FOSS community actually put RMS in jail. [2] Something that I think nobody except RMS ever seriously believed in.