The gap is closing . I think Kimi 3 is only 3 months behind the US model. It’s gpt 5.5 class model , which was released in the end of April.

It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website

> How does the prompt “Generate an SVG of a pelican riding a bicycle” add up to 95 input tokens? OpenAI’s tokenizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 tokens, suggesting there may be an 85 token hidden system prompt. It refused to leak it though.

This is quite possibly reasoning-effort prompt which is injected before the opening <think> token whenever you set a custom reasoning effort, see e.g. DeepSeek-V4 max mode prompt: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...

It’s not bad kind of expensive for 25c but if the prompt is rendered cost is much better.

I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.

The idea is not to use pelicans on bikes but a similarly random non-sensical prompts: crows on scooters, squirrels in a moon rover etc. Then pick another one for another for next cross-llm evaluation.

Do any of the vision models render the SVG and look at the result.

Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful.

Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.

My personal benchmark for new models has been to compare video making skills with something like remotion. Usually reveals if they have any "taste" or outside the box thinking.

I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.

And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one. Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.

https://mesmer.tools/benchmarks/ai-video-generation , I usually put basic ones here.

Is there a gallery of all pelicans generated by simon over time?

I would be surprised if pelican svgs are not part of the training corpus rn

Don't see why we have to have this spammed every model release when Fable class models perform the same as Opus on basic tasks like these.

Another day, another model and another pelican :-)

I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?

Imagine what amazing SVG generators we could have if Simon had randomized the target image from the start (and companies wouldn't just overfit on pelicans).

If anyone wants to try SVG generation from different models, I made this: https://codeinput.com/svg (here is an older generation: https://codeinput.com/s/5KEGl1e3rB3)

You still need an OpenRouter API Key and be careful this can burn quite a bit of money.

Imagine shilling some CLI tools no one uses in this post.

K3 is as expensive as Sonnet, not great at writing English, is handing IP back to the Chinese, and once open source will be difficult to run at scale without the compute that OpenAI and Anthropic have largely grabbed.

Sorry, how again is this the end of the frontier labs?

> This is expensive—the pelican cost 25 cents!

Engineers get unbelievably silly about evaluating costs of things.

"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.