Large Model, Small Model, or Both?

A task arrives and something decides which model runs it. In most systems that something is a constant near the top of a file, or a default inherited from whoever wrote the service first. The question is what should replace it: how a deployment should decide whether a given task goes to the expensive model, the cheap one, or some arrangement in which both take part.

The answer is that the decision is not made by rating how hard the task is. It is made of an estimate — a prediction of how well a candidate model will do on this particular task — compared against a threshold the deployment sets. Everything else follows from that: which of two available estimates is less wrong for your traffic, and whether either of them beats random assignment at the call rate you intend — the share of your traffic you mean to send to the strong model. Then what the third option of running both models together actually turns out to mean; how much there is to gain once the naive baseline and the decision's own overhead are charged; where the threshold belongs given what your failures cost; and how long the arrangement survives the models changing underneath it.

An earlier report in this series asked whether a harness could make a small model match a large one, and concluded that substitution works where a task affords a check independent of the model's own judgement, where the horizon is short, and where the problem sits in the band the small model already sometimes reaches. That instinct points at the right variable and draws the line in the wrong place. It is not that some tasks carry a check and others do not. Every task carries two possible estimates of how a model will do: a guess formed before anything runs, and a judgement formed after a cheap attempt.

Most of the measurements below come from recent preprint work, and where a finding rests on a single study the text says so at the point the finding is used.

The decision is an estimate, not a difficulty score

None of the routers described below makes its call by scoring difficulty alone. What the machinery predicts is a comparison between candidates, and what turns that prediction into an action is a number an operator chose.

The clearest example forms its estimate before anything runs. RouteLLM trains a small router to predict the probability that the strong model beats the weak one on a given query, learned from the 80,000 battles of the online Chatbot Arena platform, and turns that probability into a decision by comparing it against a cost threshold [1]. Amazon's shipped prompt router works the same way from the outside: it analyses each incoming prompt, predicts the response quality of each candidate model before generating anything, and departs from a designated fallback model only when the predicted response-quality difference clears a threshold the customer sets [2]. Two independent implementations, one shape — a scalar prediction and a knob. In neither does difficulty appear as a primitive.

The other kind of estimate is formed afterwards, by looking at an actual answer. A cascade runs the cheap model first, scores its output with a separate and much smaller scorer, and escalates only when the score falls short. FrugalGPT's published instance is concrete about the parts: on one task it queries GPT-J first and accepts that answer if a DistilBERT regression scorer rates it above 0.96, otherwise queries J1-L, otherwise falls through to GPT-4 [3]. This is the shape most people mean when they say "start cheap and hand up," and its price is exactly what it looks like — a full generation that gets thrown away whenever the cascade escalates.

These read like rival philosophies. They are not. Work that derives both from one framework treats every strategy in the space as a pair of estimators — one for quality, one for cost — with the strategy being whatever optimisation runs over them, and proves that a unified policy reasoning over both timings dominates either alone [4]. The fork between deciding up front and deciding after a first attempt is therefore not architectural. It is a consequence of which of your two estimates is less wrong.

Which estimate — and whether either one is good enough

The two estimates are not symmetric. Work that studies both timings side by side reports that the after-the-fact approach "generally outperforms pre-generation routing", because the judgement sees the actual answer, and that cascade routing "is more stable and superior" while requiring a complete generation process [5]. So the default tilt is toward attempting cheaply first — but it is a tilt with a receipt attached, and the receipt is the discarded generation plus the added latency.

The more useful correction is about where the estimate has to be right. A 2026 survey of the field reports, under the name routing collapse, a failure mode in which routers that predict a scalar performance score over-select the most expensive model as the budget rises, traced to the fact that the routing decision is a discrete comparison between near-tied candidates, so small prediction errors flip the outcome [6]. Average accuracy across the whole distribution says little about that band. The counter-move is measured, though on one workload: calibrating a token-level uncertainty signal by isotonic regression cut serving cost, measured as H100 latency, by 31 percent at a fixed micro-F1 of 0.91 on a production named-entity workload of 75,000 queries served by 4B and 12B models [7]. That paper's own conclusion is that "calibration, not threshold tuning, is the part of the pipeline worth engineering" [7].

Which leads to the precondition that ought to come before any of this. A routing estimate is worth having only if it beats a random router at your intended call rate on your own traffic, and that is a low bar which published routers have been measured to fail out-of-distribution. RouteLLM's own routers, trained on Chatbot Arena preference data, fell below the random baseline on out-of-distribution GSM8K — the worst of its four parameterisations by 11.8 percent in the paper's own metric — and climbed above random only after augmentation with roughly 120,000 GPT-4-judged pairwise comparisons over open-ended chat prompts, which were not matched to that benchmark's domain [1]. The test itself is not free, and it should be booked like every other entry here. Take a sample of your own traffic, run both models on every item in it, and decide by a standard you can state whether the small one sufficed. Then fit your estimate and score it on the items where the two models came out close rather than across the whole sample, because that narrow band is where your threshold will do its work. Only then compare it against random assignment at your intended call rate. Those labels are the bill — two generations per sampled item, plus whatever adjudicates them — and you pay it before you learn whether a router was worth building. If the estimate does not win, you have your answer, and it is not a router.

"Both together" is three different things

Both of those estimates assume one model will take the task. The third option — letting a large and a small model work together — is not a choice of the same kind. At least three unrelated arrangements travel under that label, and only two of them are allocation decisions at all.

Speculative decoding pairs a small draft model with a large verifier, and it belongs to the serving layer rather than to this decision. vLLM's documentation states that its implementation of speculative decoding "is algorithmically validated to be lossless", naming rejection-sampler convergence and greedy-sampling equality among its key validation tests [8]. The distribution the caller receives is theoretically the large model's, lossless up to the precision limits of hardware numerics [8]. Nothing about quality is being allocated, so the arrangement answers a latency question, not this one.

Planner–executor orchestration is a real division of labour, and the evidence about it is the most surprising thing in the area.

With thinking enabled the score is effectively flat across executor scale, which the authors read as evidence that such systems are planner-limited rather than executor-limited [9]. That conclusion holds inside a band rather than in general. A second controlled study varying both seats finds the multi-agent advantage disappearing once the sub-agent is a much stronger model, and summarises the pattern as multi-agent systems being "most effective at the edge of sub-agent competence" [10]. And a third, holding thinking tokens constant across three model families on multi-hop reasoning benchmarks, finds single-agent systems matching or outperforming every multi-agent architecture it tested [11]. The honest reading is that a split pays when your executor sits below competence on the work and your scaffold is not quietly buying the advantage with extra computation.

The third arrangement is the one that has become a genuine routing decision: per-step selection inside a single agent loop. A step-level cascade for computer-use agents runs a small GUI policy by default and escalates to the large model only when a stuck monitor or a milestone monitor fires, reaching, in the one pairing it reports at that operating point, 58.2 percent success on OSWorld at $0.051 per task, which the paper gives as 61.4 percent cheaper than always-large inference [12]. Its argument for why query-level routing cannot serve here is the load-bearing part: "difficulty is state-dependent and can change abruptly within a single task" [12]. That is a different decision object from the per-task one, and it prices differently.

What the prize is, and what it is charged

The headline savings are spectacular and mutually inconsistent. FrugalGPT reports cost savings of 98.3 percent on HEADLINES and 73.3 percent on OVERRULING, both measured against GPT-4, and 59.2 percent on COQA, measured against GPT-3 [3]. The spread is not a controlled comparison — the baseline changes, and the paper learns a separate cascade on each dataset's own training split [3]. What makes the saving a property of the traffic is not the spread but the condition the paper states for getting any of it: "the training examples should be from the same or similar distribution as the test examples" [3].

It is also a property of the price gap being arbitraged, and that gap has a stated origin. FrugalGPT's economics were earned against a spread of roughly two orders of magnitude between the frontier API and a hosted small model at the time it was written [3]. A router cannot save more than the gap it is trading on, so a narrowing spread narrows the prize mechanically, whatever the router does.

Against a naive baseline, less survives than the literature implies. RouterBench, the benchmark that popularised the cost-quality frontier, reports that "none of the routing algorithms significantly outperform the baseline Zero router" — a probabilistic mix of models requiring no learning at all [13]. RouterArena, built to stratify queries by difficulty, finds most routers exceeding 89 percent accuracy on easy questions with accuracy dropping sharply on hard ones, and singles out one router that reaches only baseline-level accuracy while incurring 378 percent of the cost [14]. On live agentic execution the numbers become dollars.

Routing was worth doing there only in the one configuration whose supervision came from the workload itself.

Then the decision layer charges its own rent, and two figures here pull in opposite directions because they answer different questions. The marginal per-request cost of a small router is negligible: RouteLLM measures its most expensive router as adding no more than 0.4 percent compared with GPT-4 generation [1]. The infrastructure to run one is not: dedicating one A10G per node for routing across a 32-node serving cluster adds roughly $281,000 a year, rising to roughly $323,000 on A100s at 32K tokens with SDPA [16]. That is the arrangement the paper reporting those figures exists to reject — its own compressed classifier, co-located beside the models it routes for, is booked at no dedicated GPU cost at all [16]. The first number supports the claim that the routing computation is cheap; the second supports the claim that a badly sited routing deployment is not — and neither includes the cascade's discarded first generation, which is usually the largest entry in the ledger.

The two ways of being wrong are not mirror images

The ledger has a second half: what it costs to decide wrongly. The tidy version of that asymmetry says over-routing wastes money once while under-routing loses the task. Half of that is right, and the half that is right is not right for the reason usually given.

Under-routing early does not stay local, because a trajectory has no undo. A study that routes between a mini and a full frontier model at every step of roughly twenty-step trajectories across three agentic benchmarks found that simply spending the entire large-model budget on the opening steps rivals far more complex trained policies, and reads that as underscoring "the critical importance of early-stage trajectory correctness" [17]. The blast radius has been measured directly. Forking live SWE-bench Verified trajectories, rebuilding the environment and continuing each fork with a different model, one audit found swaps rewriting 61 to 94 percent of post-fork actions [18].

A substituted step early in a run re-decides most of what follows.

But over-routing is not a single wasted call either. The same branching study observed the stronger model exhausting tight step budgets without submitting more often than the weaker one, which its authors name a possible "thoroughness tax" that budget-aware routers may need to price [18]. And the binary is missing a quadrant: on tasks neither model can solve, spending large is pure waste, which is why the budget-aware work's learned policy pulls Easy-instance spending down to 21.8 percent and concentrates 52.2 percent on Hard instances rather than distributing by apparent difficulty [17].

What makes both directions dangerous is that neither announces itself. A badly routed task returns a well-formed answer, not an error, and the machinery deployments already have does not fire on it: one agent framework's documentation is explicit that its fallback models are triggered by a model-provider error and that other exceptions propagate without trying another model [19]. Recovery, where it exists, is engineered rather than native, and it is expensive: one published repair loop fixes 13 of 73 failed GAIA tasks in a single rerun, against 4 to 6 for decoupled self-correction baselines, with strict attribution of which step caused the failure topping out at 28.8 percent [20].

One bound on the prescription this leads to. Routing up buys capability, not reliability, and the two are easy to blur. A benchmark of operational safety for coding agents in stateful workspaces reports that "safety does not simply follow from capability scaling", with one strong reasoning model showing among the highest vulnerability rates it measured [21]. Put the expensive model where the capability gap decides the outcome, and do not expect it to also serve as a safety control.

The procedure

Six steps, in order, each of which is a measurement rather than a preference.

Measure your own difficulty mix. It is one determinant of the saving, not the only one — the price gap you are arbitraging and the decision layer's own running cost, including the generation a cascade throws away, sit beside it. The sample you built to test the estimate already carries this one: read off the proportions. Nothing cited here fixes how large that sample should be, or what standard makes a small model's answer sufficient, so both are your call. The published alternatives are worse buys for most deployments: banding a query by how many of 42 models answer it correctly is rigorous and out of reach [14]. The other — a router that models each candidate's ability against each query's difficulty and reports both — is a research instrument whose own paper says plainly of its benchmark queries that they "do not cover the wide variety encountered in real-world usage" [22].

Test before you build. On that sample, check whether your intended estimate beats random assignment at the strong-model call rate you plan to run. If it does not, always calling the strongest model is the correct decision, not a failure of nerve.

Choose the timing from the sample, not from taste. If a check on the produced answer is cheap and informative for your work, attempt small first — but charge the cascade the discarded generation and the added latency before you compare. If the check is expensive or unreliable, you are betting on the pre-generation estimate, and its calibration near the boundary is the thing to engineer.

Own the threshold. Shipped routers hand you the knob and say so. The RouteLLM repository provides a calibration utility that inverts a desired share of strong-model calls into a cut-off, and warns in the same document that "the range of meaningful thresholds varies depending on the type of router and the queries you receive" [23]. What the number should mean is older than the routing literature: the deferral rule is to hand a case up when the base model's error rate exceeds the cost of deferring [24]. Substitute your own two costs; no shipped default knows them.

Spend the expensive model early, not where the task looks hard. Front-loading the budget rivals trained policies under strict caps [17], and divergence after a substitution decreases with fork depth, so the same swap does less damage the later it happens [18].

Instrument the action channel. Because a bad route returns an answer rather than an error, output checks will not see it. Watch the realised share of strong-model calls, log the model identity actually returned per request, and sample a paid counterfactual — run the alternative model on a slice of traffic — because production logs alone cannot tell you about a road not taken. Each of those is attested as observable; none is attested as catching a bad route, so treat them as instruments rather than as a guarantee.

It expires, and the calendar is published

A routing policy is fitted to a model pair and a price ratio, and both change on a schedule you do not set. The useful move is to split the policy into the half that travels and the half that does not.

The query-side estimate travels, with caveats. RouteLLM ran its trained routers without retraining over two model pairs absent from its training data and reported comparable performance, concluding that the routers "have learned common characteristics of queries that allow them to distinguish between strong and weak models" [1]. That result is reported on one benchmark, by the router's own authors. The one independent audit that carries these routers onto a pair their authors did not choose finds them making category-driven rather than difficulty-driven decisions — one BERT-based router directs all coding and mathematics queries to the most powerful model even when simpler models would suffice [25]. What transferred was partly a prior inherited from the preference corpus, not a measurement of the new pair.

The threshold does not travel, and the shipping artifacts say so in the same breath as the transfer claim: replace the model pair in configuration without retraining, then recalibrate the threshold on a sample of your own incoming queries [23]. The commercial surfaces are blunter about their limits. Amazon's documentation states that intelligent prompt routing "can't adjust routing decisions or responses based on application-specific performance data" [2]. These two positions do not resolve each other — one is about a query-side estimate generalising, the other about a shipped policy not learning from your traffic — and a reader should hold both.

The re-fit trigger is a published date. Amazon Bedrock states that a model remains available for at least 12 months before its end-of-life date [26]. Microsoft Foundry sets a generally available model's retirement date programmatically at 18 months out and exposes it through an API [27]. Anthropic commits to at least 60 days' notice of a deprecation with recommended replacements [28]. Treat each of these as a scheduled event on which the price ratio and the model pair your threshold was fitted to are known to move.

Detection between those events is harder than it looks, and there is a mechanism for why. A deployed router receives feedback only from the model it actually called: it "observes the accuracy and cost of the selected model but learns nothing about the alternatives" [29]. That is why the counterfactual has to be bought rather than mined: the unchosen arm was never observed, so no accumulation of logs will produce it. Without that purchase, a stale policy simply keeps returning answers.

Conclusion

The question "large, small, or both" turns out to be three questions wearing one coat. "Both" is mostly a category error — a serving optimisation, a decomposition and a per-step routing decision filed together — and once it is unpacked, what remains is a single decision made twice over: an estimate of how a candidate will do, and a threshold that turns that estimate into a call.

Almost everything that matters about that decision is a measurement on your own traffic rather than a fact about routers. What fraction of your work is genuinely easy decides the size of the prize. Whether your estimate beats random at your intended call rate decides whether to build anything. Which of your two estimates is less wrong near the boundary decides whether to guess up front or attempt cheaply first. And what your failures cost decides where the threshold goes — remembering that in a multi-step task the two directions of error are unequal not because the small model is worse, but because a trajectory has no undo and an early wrong step re-decides everything after it.

Limitations

The load-bearing agentic measurements here are thin. The blast-radius figures for a mid-trajectory substitution come from a single study on one scaffold, one benchmark family, one model family in two serving configurations, with thirty instances per run pair and absolute task-resolution rates in the low single digits [18]; the direction is corroborated elsewhere, the magnitude is not replicated. The same study's authors are explicit that they demonstrate the inputs to replay-based evaluation are invalid, not that published routers have been mis-ranked end to end [18] — so numbers in the literature derived by replaying logged trajectories with a substituted model should be treated as unproven rather than as refuted.

The planner-limited result rests on one controlled sweep within a single model family, reported without run-to-run variance, and a second study reverses it once the executor is strong [10]. Where the orchestrator seat should sit is genuinely unsettled: one study finds orchestrator-level thinking to be the critical ingredient [9], another finds an instruction-tuned orchestrator beating a reasoning-model one [10], and no evaluation cited above holds those two knobs in a single comparison.

Current vendor list prices could not be established from a first-party source, so only the mechanism is stated above — a router's saving is bounded by the price gap it arbitrages — and the present-day ratio is left for the reader to look up. The entry-gate test is prescribed here without a calibrated price, and the one paper that measures a router escaping the random baseline did it with generic judge-labelled chat data rather than domain-matched labels [1], so the instruction to label your own traffic is a recommendation the evidence does not settle. No study cited above ties a weaker model to more irreversible harm, and the operational-safety benchmark points the other way, finding safety not tracking model scale [21]. Nor does any of them supply a rollback or recovery procedure for a bad route once it has been taken; the evidence reaches as far as showing that recovery is engineered rather than native, and stops there.

Finally, the publication status of most of the primary work cited is unconfirmed — these are preprints, and no peer-review record was available for the majority of them. Several are recent enough that their existence has not been independently corroborated here beyond the archived documents themselves.