A Workflow Is How You Spend a Verifier

The bet everyone makes

The fashionable claim is that a compound system — several cheaper model calls wired into a workflow — beats a single expensive monolith looping on its own. We wanted to believe it. So instead of assuming it, we spent a lot of compute trying to make it true, honestly, and watched where it broke.

The short version: workflows kept tying the monolith — until one change turned every tie into a decisive win. That change tells you exactly what a workflow is for.

Four honest ties (and a few wins that evaporated)

We started on τ-bench, a multi-turn customer-service benchmark. A hand-built compound system (a per-turn router escalating hard turns to a stronger model) tied a well-prompted cheap monolith — and cost more. Then HumanEval: a cheap model already solves ~92% first-try, so there is simply no room for a workflow to add anything.

So we moved to BigCodeBench — genuinely hard (pass@1 ≈ 0.53), with real headroom: sampling five candidates, at least one is correct about +11 pointsmore often than the monolith's single try. The workflow's whole job is to pick that one. We tried two honest pickers — the model writing its own assert tests, and behavioral consensus across candidates — and both tied the monolith.

The turn: give the workflow a real verifier to spend

Both honest pickers were guessing which candidate is correct. On hard code you can't guess that reliably. So we stopped guessing and used the one thing that actually knows: the tests. We split each task's real unit tests into a public half (the agent may run them) and a held-out half (scoring only, so it can't cheat). The workflow becomes a repair loop: generate → run the public tests → read the failures → fix → repeat.

Same benchmark, same base model, same repair structure — the only thing that changed was the verifier: fake (self-generated) → real (executed tests). That single flip took the workflow from a dead tie to recovering 21 of the 49 tasks the monolith failed, at a significance level three earlier “wins” never came close to.

The point: a monolith can't spend a verifier

It is tempting to conclude “the verifier is what matters.” That's only half right, and the missing half is the interesting one. A monolith cannot use a verifier at all. It emits code once and stops. There is no second step, no place to put a test result. Hand it the exact same tests and its score doesn't move — it never runs them, because it has no loop in which to run them.

The workflow is the machinery that spends the verifier. The repair loop is the structure that turns a pass/fail signal into an edit. The verifier is inert fuel; the workflow is the engine that burns it. Neither wins alone — and the win lives in exactly one corner:

This also explains the earlier ties without any hand-waving. The self-test and consensus workflows were the same engine — a real repair/selection loop — running on fake fuel. The loop was never the problem; the signal was. Pour in real fuel and the same engine pulls away from the monolith. The workflow is necessary and the verifier is necessary; you only win with both.

Sidebar: a verifier is not a guardrail

If you come from the LLMOps world you have met this idea under a different name: guardrails. Libraries like GuardrailsAI wrap a model's output in validators — schema checks, PII filters, toxicity, value ranges, “does this JSON have a price field in [0, 10000].” It is fair to ask whether the “verifier” in this post is just a guardrail with a fancier name. It is not, and the gap is the whole point.

A guardrail checks the output against rules about outputs: is it well-formed, safe, on-brand, within bounds. A verifier checks the answer against the problem: is it correct. Conformance is necessary but never sufficient — a perfectly schema-valid, PII-free, on-brand reply can still be the wrong answer. Our repair loop does not win because the code is well-formatted; it wins because the tests pass. A guardrail would happily wave through all 49 of the monolith's failures: they were conformant, just incorrect.

The sharper cut is what each one does in the loop. A guardrail is a gate: it blocks or repairs bad output at the door, maybe re-asks once. A verifier is a signal you spend compute to satisfy — the fuel from the sections above. You do not pour a workflow into satisfying a guardrail; guardrails are cheap gates, not optimization targets. The test for which one you are holding: can your workflow spend more compute to satisfy it? If yes — repair, resample, search — it is a verifier. If it only says pass/fail at the exit, it is a guardrail.

They are complementary, not competing. A real system wants both: a verifier for correctness (does the quoted price match the source of truth; does the discount satisfy the margin rule) and guardrails for conformance (valid JSON, no leaked PII, on-brand tone). The verifier is what the workflow spends; the guardrail is what you bolt on right before send. GuardrailsAI can supply a check to a verifier — a provenance-style validator that tests a claim against retrieved context edges toward correctness — but the library is a validator toolkit, and most of what it ships are gates.

What this means if you build agents

The useful question isn't “monolith or workflow?” It's “does this task have a cheap verifier, and does my system have a loop to spend it?”

• If the task is verifiable — a test suite, CI, a type-checker, an exact-answer check — build the loop that spends it, and it will beat the monolith. That's real coding agents, and it's why they work.
• If the task is unverifiable — an open-ended conversation like τ-bench, where there is no cheap oracle for “did this turn go well” — a workflow has nothing to spend, and it ties the monolith. Don't pay for structure you can't fuel.

This is the lens we build AutoAW around. A workflow's value is converting a verifier into quality; evolution's job is to discover the loop that spends yours best. Point it at a verifiable task distribution and it finds structure that beats the monolith. Point it at an unverifiable one and it will — honestly — tell you the monolith is fine.

How we kept ourselves honest

The held-out split matters: if the agent could run the exact tests it's scored on, it could special-case them and “pass” without being correct. Splitting the suite — public to repair against, held-out to score — keeps the win real; both are genuine test execution, just partitioned. We validated every split by confirming the reference solution passes both halves. And every headline number is a paired comparison over 150 tasks with the cache off, reported with its significance — because on noisy benchmarks, a single run is a data point, not a finding.