hmm, this is like pass@n until you get the high watermark? How would this mean anything?
I’m pretty excited to see what sort of generalization we come to over the next 12 months on the harness side: if it turns out this can be RLed in as ‘consider if building a world model might help here’ and we get this as another native capacity, that will be interesting. If we get 100 of those problem-solving strategies all included, feels like we will see another hurdle cleared in terms of usefulness.
A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.
it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful.
what this harness does is get the model to write a simulator first, it's measuring something entirely different.
the harness basically outsources the alien nature of what the LLM is asked to do to algorithms it writes (LLM doesn't really process the input, it writes scripts to chew over it and tell it the results which are easier for it to process). this would actually be impressive if you got it to do that for a much more complicated game than Arc.
> simulator the model builds is comparable to the mental model of the game humans create
Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.
We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.
Impressive results. Will this translate to coding agents (and training general purpose and for coding LLMs) too?
---
> When Michelson and Morley could not detect the medium light was supposed to wave in, Lorentz took the first route: keep the aether, patch the rules with contraction hypotheses that absorbed the null result. Einstein took the second: in special relativity, he discarded the aether as part of the state and made simultaneity frame-relative, yielding a simple electrodynamics of moving bodies.
BECs, SVT, Superfluid Quantum Gravity
Massful photons are modeled with Proca fields. Like Einstein, Proca was also a student of Minkowski. The Mass-Equivalence principle ~~does not~~ still holds if photons have mass.
(edit) Energy-momentum relation: https://en.wikipedia.org/wiki/Energy%E2%80%93momentum_relati...
> could not detect the medium light was supposed to wave in,
Superfluid Quantum Gravity (Fedi,) says that there is a medium that light waves through; there is not nothing in space, space is a quantum dilatant superfluid with near-zero viscosity.