I love the idea. @Edgee.ai we are tracking cost in real time by tag, LLM, ...but not yet on forecast, and indeed it will be very useful. Something to explore; thanks for the feedback.

One underlooked source of variance: unstructured system prompts causing output format failures, which trigger retries or correction loops.

When the model gets prose instructions, it guesses what's a constraint vs. an example vs. an objective. That guess is inconsistent across calls. Each wrong output is another LLM call.

Typed blocks reduce this. Role in one section, constraints in another, output_format explicitly tagged. The model gets unambiguous signal and output variance drops. Fewer retries, more predictable token spend.

I built github.com/Nyrok/flompt around this: decomposes prompts into 12 semantic blocks, compiles to Claude-optimized XML. Doesn't solve full cost forecasting but cuts the "bad output -> retry" part of the variance.

local models are better in controlling costs rather commercial models are very high and no control on this cost..how ever again local models training setup to be archietected very well to train this continoulsly

It feels like the traditional fixed SaaS pricing model is slowly shifting toward more consumption-based pricing.

Honestly, if you're designing your agent workflows properly with hard limits on retries and tool calls, the variance shouldn't be that wild. Most of the unpredictability comes from not having those guardrails in place early on. A few weeks of real production data usually shows the average cost is more stable than you'd expect.

imo switch to local models could be an option

Just add very hard high limits and add instrumentation so you can track it and re-evaluate it accordingly.

This takes a couple of hours maximum at best.