Model routing thoughts by AI

Analysis following topic base on materials available from internet in recent two or three months:

It seemed that multiple model service providers are implementing smart/intelligence-level-based model routing in their backend now a days. There's already some level of similar feature embedded in coding clis like claude-code, gemini-cli, opencode, etc.

What is the best place to implement model routing from your opinion? What is the best way to implement it considering about user friendliness, trusting, user-s-cost, user-experience, performance and accuracy.

 I'll check what has been published recently on this first.

What the recent material shows

Routing now happens at several layers, and the layers behave quite differently.

• Gateways and routers: LLM Gateway added "smart routing" in late September. The organization picks up to 30 candidate models and a classifier, and it does not quietly fall back to a model you did not approve. Hosted routers such as OpenRouter add roughly 25 to 50 ms of overhead.
• Inside the product: One Cursor user describes the "Auto Balanced" router as a learned model that reads your recent conversation, predicts how well each candidate will do and what it will cost, then picks within a budget.
• Client-side routers: A community package for coding harnesses persists every routing decision with its reasons so it can be replayed, and demotes cheap-but-flaky models using escalation and error rates from your own traffic.
• Harness to backend hints: Codex merged a change in August that sends an x-codex-routing-hint header with the model and service tier, only on Codex-backend authentication. This suggests the client and the backend are starting to share routing signals.

The recurring problems are about trust, not technique:

• A user on OpenAI's forum reports requesting a thinking model and being served a mini model.
• An OpenRouter issue describes an "opus-latest" alias silently walking down to older models that were billed at a higher price.
• A Cursor user asked for a justification of each routing choice after a session jumped unexpectedly to Opus 5.
• Reported savings of up to about 85% only hold with a quality floor. Without one, cost routing silently degrades output and nobody logs it.
• A Hacker News commenter objected that a proxy breaks the control loop. The router doesn't know that a cheap model just failed, and it may not know that re-fetching context with a smaller model can cost more than staying on the parent.

My opinion: where it should live

I'd split the decision by who holds the information, rather than pick one place.

1. The agent harness (the CLI or app) should own the decision. It sees the task phase, tool results, test failures, context size, cache state, and whether you retried or undid something. That lets it escalate on actual failure instead of only predicting difficulty up front. It can also show you what it chose and let you override it.
2. The provider should supply signals and safe levers, not silently swap models. It knows fleet load, pricing, and capabilities. It can offer things that don't change the model you picked, such as effort or thinking budget, serving tiers, and a clearly labelled "auto" alias. Swapping the model behind an explicit selection is where the trust complaints come from.
3. A gateway is best for organization policy. Budgets, allowed-model lists, failover, audit, and compliance fit there. It is blind to the agent loop, so it shouldn't make the fine-grained quality calls.

One caveat: when the harness and the provider are the same company, as with Claude Code, the harness-versus-backend split matters less. The trust rules below still apply.

How I'd implement it

• Make auto opt-in and separate. Use a distinct "auto" choice. Never reroute a model the user explicitly selected. Report the model actually used, and bill by that model, in every response and usage log.
• Route by phase and role first. Planning and hard debugging go to a strong model. Search, mechanical edits, summarization, and subagents go to a cheap one. This is more reliable than classifying every prompt in isolation.
• Use cascades with verification. Go cheap-first for cheap, reversible work and strong-first where a retry is expensive. Escalate on concrete signals such as failing tests, tool errors, or user retries.
• Set a quality floor and give the user one policy knob (cost, balanced, or quality), plus a budget and hard capability constraints like context length and tool support.
• Be cache-aware. Switching models mid-session loses the prompt cache. Switch at natural boundaries, and count a full re-dispatch as a real cost of escalating.
• Keep the router fast. Use heuristics first, then a small classifier, and call an LLM classifier only when uncertain. It should add tens of milliseconds, not seconds.
• Make decisions explainable and overridable. Show a one-line reason, keep a decision log, and let the user pin a model per turn.
• Measure cost per solved task, not per token. Sample shadow comparisons against the strong model, track escalation and retry rates, and refresh the router as models change, since a hard-coded model list goes stale quickly.
• Mind privacy. A third-party router sees every prompt and your code, so it needs the same trust level as the model provider.

How the placements compare

Criterion	Harness	Provider backend	Gateway
Accuracy	Best (rich signals, closed loop)	Good if opt-in	Limited (blind to the loop)
Trust	High if transparent	Weakest if silent	Good if policy is yours
Cost savings	High (cache- and phase-aware)	High (fleet view)	Medium to high
Latency	Lowest	Low	Adds a hop
User friendliness	Best, with override	Simple, but opaque	Needs setup

Caveats

Much of this material comes from vendors, community forums, and GitHub issues, and some of the benchmark claims (such as router leaderboards) are self-reported. I found little independent evidence about how the major labs' own backend routers work, so my recommendations on that layer rest on the reported user experience, not on documented internals.

In short, the client decides, the provider exposes signals and an honest "auto" mode, and the organization sets guardrails at the gateway. The router should always disclose what it chose and why.