What 4.1 billion routed requests taught us about model choice

Across a full quarter of production traffic, 71% of prompts cleared their workspace's own quality floor on a model costing roughly a twentieth of the one they had been sent to. That number held across every industry we looked at.

The shape of the data

Between April and June we routed 4.1 billion requests for 2,400 workspaces. Every one carries a decision record: the quality floor in force, the profile scores for each eligible model, and which one won. Because customers set their own floors, we are not grading models against our opinion of quality — we are grading them against what each team already said was good enough.

The distribution is lopsided in a way that surprised nobody who has looked at their own logs, and everybody who has not. A small tail of genuinely hard prompts needs the strongest model available. The bulk does not come close.

Where the money actually goes

Three categories accounted for most of the waste we saw in the first week of each new workspace.

Autocomplete and inline suggestion

These fire on a keystroke cadence, which means volume two orders of magnitude above anything else in the product. Sent to a reasoning model, a feature used by a fraction of your users will outspend your entire batch pipeline.

Classification dressed up as generation

A surprising share of prompts are asking a model to pick one of five labels, phrased as a paragraph of instructions. The output is four tokens. The input is nine hundred.

Retries nobody counted

Hand-rolled retry loops that fire three times against the same expensive model on a timeout, rather than once against a cheaper one that is currently healthy.

The cheapest optimisation available to most teams is not a better prompt or a smaller model. It is noticing that one feature is responsible for the bill.

What we changed because of it

Two things. First, new workspaces now get a routing report after their first 10,000 requests that ranks features by spend and flags any where the median prompt would have cleared the floor on a cheaper lane. Second, the cache now recognises near-duplicate prompts rather than exact matches only, which turned out to matter enormously for classification traffic.

What we did not change

We still do not set your quality floor for you, and we do not intend to. A floor that is right for a support draft is wrong for a medical summary, and the only people who can make that call are the ones who own the feature. What we can do is make the consequences of the number you pick visible on the same day you pick it.

Reading your own numbers

If you want to run this analysis on your own traffic, the decision log is exportable as newline-delimited JSON from any workspace, including on the free plan. Group by feature, sum the cost, then look at what share of requests in your most expensive feature scored above your floor on the cheapest eligible model. In our experience that single query pays for the afternoon.