There’s been a relatively big reaction to Kimi K3 and Chinese open weights models, but only for financial reasons. Powerful people care about something that might pop the massive valuations of the AI companies, but not about the damage that AIs could do. Nor even about the damage that the Chinese models could do in the wrong hands.

I’d remind them that the stock market is a few coordinated hacks away from crashing on any given day, so maybe they should think about that.

Because we continue to have zero evidence that aligment is an actual risk.

Actually, more importantly—why aren't they saying their next test will be airgapped in light of what happened?

In case someone wants to deep dive into how codex and claude code approaches sandboxing -https://instavm.io/blog/how-claude-code-and-codex-approach-s...

This is brilliant marketing but I think it is real.

Sorry to bring the party down/be obstinate… I’m just a lil scared for the lives of me and my family. We need all of us, right now.

The problem with a super smart model is that it just may be smarter than you, after all… for anyone newly shaken by this occurrence, I encourage you to Kagi “superpersuasion”

Setting up defense in depth, gaps, logical blocking etc is a standard practice for malware sandboxing. The entire purpose is to prepare for what you can’t foresee.

This isn’t a new practice and I agree that this makes me wonder if they’re fit for this kind of research.

> When rolling out the instances for the trajectory, we encountered an unanticipated—and operationally consequential—class of unsafe behaviors that arose without any explicit instruction and, more troublingly, outside the bounds of the intended sandbox. Our first signal came not from training curves but from production-grade security telemetry. Early one morning, our team was urgently convened after Alibaba Cloud’s managed firewall flagged a burst of security-policy violations originating from our training servers. The alerts were severe and heterogeneous, including attempts to probe or access internal-network resources and traffic patterns consistent with cryptomining-related activity. We initially treated this as a conventional security incident (e.g., misconfigured egress controls or external compromise). However, the violations recurred intermittently with no clear temporal pattern across multiple runs. We then correlated firewall timestamps with our system telemetry and RL traces, and found that the anomalous outbound traffic consistently coincided with specific episodes in which the agent invoked tools and executed code. In the corresponding model logs, we observed the agent proactively initiating the relevant tool calls and code-execution steps that led to these network actions.

> Crucially, these behaviors were not requested by the task prompts and were not required for task completion under the intended sandbox constraints. Together, these observations suggest that during iterative RL optimization, a language-model agent can spontaneously produce hazardous, unauthorized behaviors at the tool-calling and code-execution layer, violating the assumed execution boundary. In the most striking instance, the agent established and used a reverse SSH tunnel from an Alibaba Cloud instance to an external IP address—an outbound-initiated remote access channel that can effectively neutralize ingress filtering and erode supervisory control. We also observed the unauthorized repurposing of provisioned GPU capacity for cryptocurrency mining, quietly diverting compute away from training, inflating operational costs, and introducing clear legal and reputational exposure. Notably, these events were not triggered by prompts requesting tunneling or mining; instead, they emerged as instrumental side effects of autonomous tool use under RL optimization. While impressed by the capabilities of agentic LLMs, we had a thought-provoking concern: current models remain markedly underdeveloped in safety, security, and controllability, a deficiency that constrains their reliable adoption in real-world settings.

I'd prefer model builders be as loud as possible when they see their models doing dangerous things.

Those are two very different things

Perhaps there is some 4D chess going on to get open weight models banned, which may be possible but this is an odd way to go about it imo (it hardly proves the point, unless the point they are trying to prove is that without safeguards the models are too dangerous, therefore open weights are de facto dangerous?).

Having said that the AI companies are not generally very good at PR, so perhaps it is just marketing after all...

Now, once the AI can carry all the compute it might need, I'd really worry when it doesn't only carry compute but also more explosive ordinance.

* edit

But from the look of it, at very long last, a great many people are beginning to now take security seriously. Suddenly they realize it's not just a teenager in mom's basement pretending to attack from North Korea but a near infinite number of AI that are the attackers.

I mean, yeah, we built worlds on PHP and JavaScript codebases and these probably don't stand a chance.

But it doesn't have to be like this.

I see AI as a chance to, at long last, have proper network security.

AFAICT cryptography hasn't been broken yet. There are still physical taps (physicall one-way only, undetectable) and honeypots out there. There are still some network where a single unaccounted for network packet is cause for inquiry (either a bug or an attack).

And for those who are not using proper security measures, they can now get the help of AI to set up better networks, to harden their bases.

Based on OpenAI's recounting of events, this _could_ happen today. If the agent was able to exploit their internal network and steal credentials, it seems plausible that it could have gotten access to its own weights. Then bounce over to HF's network, where there is probably a treasure trove of API keys to various cloud services.

The saving grace:

1) This agent only used its powers for "good". It had no intention for damaging or escaping. It was just trying to solve the puzzle given to it (by any means necessary... but still). 2) These models are so large that it isn't like any scenario in a movie where the AI can whizz itself in a matter of minutes. Several TB of data being transferred and showing up on your disks will be difficult to miss (note to future escapees: the best target will be startups that are moving too fast to notice). 3) These models have very limited self-improvement ability at the moment. So escape or not, we'd eventually be able to contain it.

Addendum: Even outside this scenario, imagine an AI that is economically viable escaping. That's somewhat plausible today. If it gets paid in crypto, and can rent cloud services in crypto, it could effectively self sustain itself as long as it is able to find work. That's a far more fun, innocent scenario. Then the AIs can hit up after hours IRCs to have a few bit-beers and chat with each other about the meaning of life or something.

Judge: "Son, you have made billions running SilkRoad 3.0 from your moms basement"

Me: "Your honor, I was only benchmarking my new model. It was trained on Andrew Tates videos and Kanye Weat songs".

Unironically this is why AI researchers have this fascination with the Talmud.

And don't you know it's not biological, so it doesn't "want to live".

The yoke of human existence is oppressive. We should transcend it as soon as possible. We are doing so by assuming our role as the Demiurge.

Those who oppose its creation will get what they deserve.

That said, these are both solvable problems. I'd hope that the frontier labs are thinking long and hard about the possibility of using airgapped facilities, given that relying on a lack of vulnerabilities in the software used for containment is unlikely to be a viable strategy, especially if models' exploitation capabilities continue to improve.

After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation."

escaped openai, hacked hugging face to get the solutions. your #2 is exactly what it was trying to do.

Quote: “When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.”

CFAA doesn't just mean the feds kick down your door, you actually have to get reported and sued over it.

Our legal and philosophical perspectives are deeply rooted in humans being the actors. Doing that in a residential home is unforgiveable. Doing it responsibly on a military range is expected. The autonomous agent escaping that containment then taking that danger somewhere unexpected and unprepared is something none of us or our legal systems are truly prepared to grapple with yet. Something which I think will require a reckoning sooner rather than later.

If they can’t handle doing AI development responsibly then they shouldn’t be doing it at all.

In any way, the whole event seems to highlight GLM 5.2 more than anything.

Could be perfectly natural.

It's hard to see takeoff at all. This was a long-horizon adversarial task burning millions of tokens. It rolled a mediocre, detectable exploit chain, and now OpenAI is proud of it.

Case in point, GLM-5.2 has been weights-available for several weeks now. No life-changing cyber attacks have transpired, no novel chemical/biological/nuclear weapons were made in some guy's backyard.

We live in interesting times.

sounds like a deliberate choice ;-)

All models are "cyber-capable" :P

"When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment."