A Cheaper, Careful Agent Beats the Expensive Monolith

The monolith and the compound system

The path of least resistance for an agent is what the literature calls a monolithic agent: a single, capable, expensive model handed the full toolset, looping until the task is done. It works, and it is simple. It is also the most expensive thing you can run — you pay frontier-model prices on every turn, including the many turns that are trivial.

The competing thesis — Berkeley calls it the compound AI system — is that a structured arrangement of cheaper parts can land on a better point of the cost/quality frontier. Our question is the sharpest version of that bet: can one cheap, carefully-prompted reasoning model beat one expensive model with all the tools? Cheaper is nearly guaranteed. The interesting claim is not worse — possibly better.

The arena: τ-bench retail

We run this on τ-bench (retail) — a multi-turn customer-service benchmark where an agent talks to a simulated customer and calls real tools against a store backend. It is a good arena for this question because it has the structure the compound thesis exploits: most turns are cheap lookups (get_order_details, find_user_id_by_email), while a few are irreversible, policy-gated writes (cancel_pending_order, return_delivered_order_items, modify_pending_order_*). The monolith pays frontier price on the easy turns; the score is decided on the hard ones.

The contestants

Result 1: cheap and careful nearly matches, for 1/24th the price

We ran all three in Benchmark mode on the same 20 retail rows — a fixed config scored across the slice, no evolution. To get honest error bars we disabled the response cache and ran multiple independent passes (the only randomness in τ-bench is the simulated customer, at temperature 0.3). Quality is τ-bench task success; cost is USD recomputed from tokens at list price.

That is the headline: a sharp prompt on a cheap reasoning brain keeps roughly three-quarters of the frontier model's task success while cutting the bill by more than an order of magnitude. If cost is anywhere in your objective, B is the configuration to start from — not the monolith. (One caveat we'll own: AutoAW picks the simulated-customer model by provider, so the gpt-4o run is simulated by gpt-4o-mini and the Nebius runs by Llama-3.3-70B. The 24× cost gap dwarfs any customer-model effect, but the exact quality delta carries that asterisk.)

Result 2: the compound bet that didn't pay off

τ-bench looks tailor-made for a compound system: most turns are cheap lookups; the score is decided on a handful of irreversible writes. So we taught the runner a per-turn difficulty router — read the conversation, and escalate only the high-stakes turns to a stronger model. The mechanism works exactly as designed: on a return, the router keeps the cheap model for the lookups and switches to the strong one for the return decision and its confirmation.

Our first version made things worse than the plain cheap agent. The culprit was a subtle one: the router had an “answer the customer myself” option for trivial turns, and it kept using it on turns that actually needed a tool call. It would reply “Okay, I've noted that” — and the required write never executed. The cheaper turns weren't cheaper; they were empty. Removing that option, so every turn goes to an agent that can act, fixed the regression.

And then the interesting part. The fixed compound ran three passes and beat the cheap agent by 16% (0.356 vs 0.306). Exciting — and wrong. Three more passes reversed it (0.289 vs 0.367). Pooled over six passes, the compound and the cheap monolith are a statistical tie (0.322 vs 0.336, about 0.4σ apart) — and the compound costs 50% more and swings harder. τ-bench's 20-row variance is large; a three-run “win” of a few points is noise wearing a result's clothes.

Result 3: the lever that moved was the model, not the wiring

If hand-built structure can't beat the cheap agent, the other axis is the model itself. So we stopped tuning architecture and swept it instead: the same careful single-agent config — same tools, same policy prompt, same slice — run across a range of models, cheap to strong. The question was blunt: is there a cheaper-but-good-enough model than gpt-oss-120b@low?

The frontier collapses to one model at two effort settings. gpt-oss-120b@low — that is agent B, and its 0.30 here versus 0.34 earlier is the same 20-row noise we keep meeting — is the cheapest point; turning the reasoning dial to @high buys a few points of quality for ~1.7× the cost, and even that gap is inside the noise. Everything else is dominated: lower quality and, mostly, more expensive.

Two of the losers deserve a footnote. DeepSeek-V4-Pro tops chat and reasoning leaderboards, yet it landed near the bottom here at 3× the cost — capability on open-ended reasoning does not automatically transfer to disciplined, stateful tool use. And Llama-3.3-70B was the worst and the priciest by 10×: it churns tool-call iterations without terminating, burning ~100k output tokens a conversation. A low per-token price means nothing if the model won't stop.

Two ways we almost fooled ourselves

Both are worth stealing for your own agent benchmarks. First, the cache lies about variance. A warm response cache makes repeated runs bit-identical replays — three “passes” that are really one, with zero spread. Cost stays honest (it's recomputed from tokens at list price), but any claim about run-to-run stability is meaningless until you turn the cache off. Second, route the model deliberately. Our expensive baseline first scored near-zero because gpt-4o was silently routed to a provider that doesn't serve it and 404'd every turn — a “result” that was really a plumbing bug. Read a transcript before you trust a number.

Reproduce it in AutoAW

The core comparison is two benchmark runs on the same dataset slice:

1. New Experiment → Benchmark. Pick “Benchmark a fixed config” and select a τ-bench retail dataset. AutoAW sets the runner to tau-bench. Fix a dataset slice so both runs are comparable.
2. Run B — the cheap agent. Click “Use recommended seed”. That attaches openai/gpt-oss-120b@low with the policy-disciplined prompt and pins the provider to Nebius. Start the run.
3. Run A — the monolith. Start a second benchmark on the same slice, seeding a single agent on gpt-4o with the full toolset and a generic prompt.
4. Compare — and run each more than once. Read off quality, cost, and latency side by side on the leaderboard. Then repeat each run a few times: one pass is a data point, not a finding.

Takeaways

• A cheap reasoning model with a disciplined prompt kept ~74% of gpt-4o's quality at ~1/24th the cost. Start from the cheap careful agent, not the monolith.
• A hand-built compound system did not beat that cheap agent here — it tied, at higher cost. Structure is not a free win.
• Sweeping the model axis was more decisive than any architecture: gpt-oss-120b@low already sits on the frontier, and no cheaper model was good enough — stronger names like DeepSeek-V4-Pro and Llama-3.3-70B came in worse and pricier. Search the model before the wiring.
• A “win” from three runs vanished under six. Disable the cache and run multiple passes before you believe a small quality gap.