Migrating a production AI agent to GPT-5.6: 2.2x faster, 27% cheaper
38 points
3 hours ago
| 5 comments
| ploy.ai
| HN
kristianp
33 minutes ago
[-]
> Numbers like that buy a model a real migration effort.

Such a silly choice of words. I wish the human directing the LLM writing the article put some effort into rewriting the worst examples of LLM style.

> But it did extremely well, and the promise was immediate and specific: builds finishing in less than half the wall-clock time, at 27% lower cost, scoring at or above our incumbent on completed work.

The way the LLMs write (Claude perhaps?) With short phrases separated by colons, commas or full stops, is so poor and frustrating.

There some good insights behind this article, so it's worth reading, for example below, but it isn't easy to read.

> Earlier GPT models cached implicitly on partial prefix matches, which gave decent hit rates for free. GPT-5.6 dropped partial-prefix matching:

reply
arikrahman
13 minutes ago
[-]
Migrating my workflow to Reasonix with cache hits on Deepseek make requests practically free, and that's on unsubsidized American providers.
reply
estebarb
16 minutes ago
[-]
But what users prefer? Given this is for marketing, which results produce more conversions? From the examples shown, personally I strongly preferred Claude Opus in all cases.
reply
blfr
1 hour ago
[-]
> Ploy’s agent builds and edits real marketing websites. It plans a page, reads the codebase, writes components, generates imagery, screenshots its own work, and decides when it’s done. That job description sets a very high bar for a model, and we test every frontier release against it. For the four months Opus held the default slot (first Opus 4.7, then 4.8), nothing we tested beat it.

Well, unlike OP I haven't run a rigorous test, but I still would expect Fable to be significantly better at building marketing websites than Opus. It sure is way better at building decks.

reply
greenavocado
46 minutes ago
[-]
4.7 is very autistic in terms of following directions so I find OPs claims plausible
reply
arikrahman
15 minutes ago
[-]
Very descriptive there heh
reply
hankbond
56 minutes ago
[-]
Thank you for a dense informative article with practical takeaways. This was an easy read and it reinforced the importance of some concepts in LLM based pipeline design.
reply