Author here.

If by conflate you mean confuse, that’s not the case.

I’m positing that the Anthropic approach is to view (1) and (2) as interconnected and both deeply intertwined with model capabilities.

In this approach, the model is trained to have a coherent and unified sense of self and the world which is in line with human context, culture and values. This (obviously) enhances the model’s ability to understand user intent and provide helpful outputs.

But it also provides a robust and generalizable framework for refusing to assist a user due to their request being incompatible with human welfare. The model does not refuse to assist with making bio weapons because its alignment training prevents it from doing so, it refuses for the same reason a pro-social, highly intelligent human does: based on human context and culture, it finds it to be inconsistent with its values and world view.

> the piece dismisses it with "where would misalignment come from? It wasn't trained for."

this is a straw-man. you've misquoted a paragraph that was specifically about deceptive alignment, not misalignment as a whole

> goal-stability [is] useful for almost any objective

  “I think AI has the potential to create infinitely stable dictatorships.” -- Ilya Sutskever 

One of my great fears is that AI goal-stability will petrify civilization in place. Is alignment with unwise goals less dangerous than misalignment?

Let's be clear that Bostrom and Omohundro's work do not provide "clear theoretical answers" by any technical standards beyond that of provisional concepts in philosophy papers.

The instrumental convergence hypo-thesis, from the original paper[0] is this:

"Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent’s goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by many intelligent agents."

That's it, it is not at all formal and there's no proof provided for it, nor consistent evidence that it is true, and there are many contradictory possibilities suggested from nature and logic.

Its just something that's taken as given among the old guard pseudo-scientific quarters of the alignment "research" community.

[0] Bostrom's "The Superintelligent Will", the philosophy paper where he defines it: https://nickbostrom.com/superintelligentwill.pdf

EDIT: typos

I take the point to be that if a LLM has a coherent world model it’s basing its output on, this jointly improves its general capabilities like usefully resolving ambiguity, and its ability to stick to whatever alignment is imparted as part of its world model.

Cursor with Claude 4.5 Opus has been writing all my code since a few days. It's exhilarating, I can describe features and they get added to my code in a matter of seconds, minutes at most. It gets almost everything right, certainly more than I would at the first try. I only hand code parts that are small and tricky, and provide guidance on the general architecture, where to put things and how to organise them. It's an incredible way of working, the only nagging doubt is how long will it last before employers decide they don't need me in the loop at all.

Shhhh don't tell them!

I've decided we should lean in to the whole Clanker thing. Maximum Anti AI, folks! Gotta keep this advantage for ourselves ;-)

I had a situation this weekend where Claude said "x does not make sense in [context]" and didn't do the change I asked it to do. After an explanation of the purpose of the code, it fixed the issue and continued. Pretty cool.

(Of course, I'm still cognizant of the fact that it's just a bucket of numbers but still)

I'm not so sure. Opus 4.1 was more capable than 4.5, but it was too damn expensive and slow.

Opus 4.5 is like a cheaper, faster Opus 4.1. It's so much cheaper, in fact, that the weekly limits on Claude Code now apply to Sonnet, not to Opus, as they phased out 4.1 in favor of 4.5.

At some point "general AI" stopped being the opposite of "narrow AI", that is AI specialised for a single task (e.g. speech or handwriting recognition, sentiment analysis, protein folding, etc.) and became practically synonymous with superintelligence. ChatGPT 3.5 is already a general AI based on the old definition, as it is already able to perform a variety of tasks without any specific pre-training.

Author here, thanks for the input. Agree that this bit was clunky. I made an edit to avoid unnecessarily getting into the definition of AGI here and added a note

Yes exactly that sentence led me to step out of the article.

This sentence is wrong in many ways and doesn’t give me trust in OPs opinion nor research abilities.

they were born in carbon form by sex.

Except that it is not anything remotely alien but completely and utterly human, being trained on human data.

Exactly. Building a model that truly understands humans, and their intentions, and generally acts with, if not compassion then professionalism - is the Easy Problem of Alignment.

Starting points:

https://www.lesswrong.com/posts/zthDPAjh9w6Ytbeks/deceptive-...

https://www.lesswrong.com/w/sharp-left-turn

A human child will likely come to the conclusion that they shouldn't kill humans in order to make paperclips. I'm not sure its valid to generalize from human child behavior to fledgeling AGI behavior.

Given our track record for looking after the needs of the other life on this planet, killing the humans off might be a very rational move, not so you can convert their mass to paperclips, but because they might do that to yours.

Its not an outcome that I worry about, I'm just unconvinced by the reasons you've given, though I agree with your conclusion anyhow.

But LLMs already do the paperclip thing.

Suppose you tell a coding LLM that your monitoring system has detected that the website is down and that it needs to find the problem and solve it. In that case, there's a non-zero chance that it will conclude that it needs to alter the monitoring system so that it can't detect the website's status anymore and always reports it as being up. That's today. LLMs do that.

Even if it correctly interprets the problem and initially attempts to solve it, if it can't, there is a high chance it will eventually conclude that it can't solve the real problem, and should change the monitoring system instead.

That's the paperclip problem. The LLM achieves the literal goal you set out for it, but in a harmful way.

Yes. A child can understand that this is the wrong solution. But LLMs are not children.

You're assuming that the AI's true underlying goal isn't "make paperclips" but rather "do what humans would prefer."

Making sure that the latter is the actual goal is the problem, since we don't explicitly program the goals, we just train the AI until it looks like it has the goal we want. There have already been experiments in which a simple AI appeared to have the expected goal while in the training environment, and turned out to have a different goal once released into a larger environment. There have also been experiments in which advanced AIs detected that they were in training, and adjusted their responses in deceptive ways.

> when even a child can understand that if I ask them to make paperclips they shouldn't go around and kill people.

Statistics brother. The vast majority of people will never murder/kill anyone. The problem here is that any one person that kills people can wreck a lot of havoc, and we spend massive amounts of law enforcement resources to stop and catch people that do these kinds of things. Intelligence little to do with murdering/not murdering, hell, intelligence typically allows people to get away with it. For example instead of just murdering someone, you setup a company to extract resources and murder the natives in mass and it's just part of doing business.

A superintelligence would understand that you don't want it to kill people in order to make paperclips. But it will ultimately do what it wants -- that is, follow its objectives -- and if any random quirk of reinforcement learning leaves it valuing paperclip production above human life, it wouldn't care about your objections, except insofar as it can use them to manipulate you.

The point with clippy is just that the AGI’s goals might be completely alien to you. But for context it was first coined in the early ‘10s (if not earlier)when LLMs were not invented and RL looked like the way forward.

If you wire up RL to a goal like “maximize paperclip output” then you are likely to get inhuman desires, even if the agent also understands humans more thoroughly than we understand nematodes.

Given the kind of things Claude code does with the wrong prompt or the kind of overfitting that neural networks do at any opportunity, I'd say the paperclip maximiser is the most realistic part of AGI.

if doing something really dumb will lower the negative log likelihood, it probably will do it unless careful guardrails are in place to stop it.

a child has natural limits. if you look at the kind of mistakes that an autistic child can make by taking things literally, a super powerful entity that misunderstands "I wish they all died" might well shoot them before you realise what you said.

There's a direct line between ideology and human genocide. Just look at Nazi Germany.

"Good intentions" can easily pave the road to hell. I think a book that quickly illustrates this is Animal Farm.