This would not work in the way that shows any significant genuine benefit IMO. Caching and optimum routing of a single request are at odds with each other. Higher the distinct model count in a conversation, more cache misses you accept.

Based on what OP said elsewhere in the discussion "threshold to switch to another model will be higher" means that essentially you reduce the workflow into two models at most. The two model primitive, one planner and one executor, is already sufficient for such a use case.

For lower than 2 models, it's just a simple single model cache-preserving conversation which arguably doesn't need another layer. For larger than 2 models, you are likely paying a large aggregate cache penalty that negates most of the gains

It's rather hard to do at the proxy level with agentic coding, such as Claude Code or similar. These are long-chained sessions of tool use that heavily rely on prompt caching. Changing mid-flight is costly.

It looks like much more context is required to decide on the best model (e.g., summarizing logs might use a cheap model, whereas you likely want Opus/Mythos/GPT 5.6 to debug multithreading logic). In an agentic system, a decision about the model may be embedded in the decision to orchestrate the model.

Wont this kill the kv cache?

Man, I'm not so sure if I'd use something like this because the way I prompt already changes based upon what model I am using. I'm not convinced it would route to the right model based on my diction or whatever.

The thing I do not get with these routers is that you will have more cache misses (5min ttl). And if there is one thing i’ve learned; using the cache is crucial.

How does this router translate to $$$ when developing?

I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session.

I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .

This + making sure common requests are saved as reusable skills and scripts would probably save a large part of my token usage

As prices increase we will see more of these tools to optimise and make the best use of token budget

What about request caching? If you swap to a cheaper model mid execution it might cost more that to make multiple requests to the already cached provider?

"We reward the routing model when it selects an LLM that achieves the task successfully" sounds pretty oversimplified

This is cool!

Will this use my Claude Pro/Max subscription? Or will it always use the API billing "pay as you go"?

I would rather just use OpenCode - leverage AI models, even can host locally or paid ones with ease.

Cool.. but I still don't get how this is going to save money. It seems to me that it might actually burn more money just because the whole system now seems to be coming from different LLMs.

Also, small LLMs are prone to stop before completion, throw errors and produce loops. Is this factored in the design of the tool? I am not sure.

edit: spellcheck

We have created Murmur[1] which kind of works with your existing subscription (having API key is not mandatory). You can just tag @copilot @codex from claude code to delegate work to them. (it can also do it on its own too btw)

1. https://github.com/instavm/murmur - Murmur

It is funny. We are building something similar.

What is the difference from Cursors 'auto' mode?

but this means you work with API pricing rather than subscription pricing. Isn’t it better to use claude or codex CLI etc directly in terms of cost?

> At Weave, we write ~all our code with AI

This is probably not a very effective way of marketing imo. At least, it turns me completely off.

Isn't this more expensive than always using the same model, since, as I understand, by routing to different models you give up on cache?

This is basically what I need, a router. I’m tired of changing intelligence & speed levels manually.

How come data privacy and confidentiality is not an issue with services like these?

Do people voluntarily use these proxies/routers, knowing their prompts, outputs and code will be seen by other people ?

I get it might be ok for personal projects, but for anything that makes money and is a part of business... this must be big no-no ?