Does MCP Search Actually Work? We Measured It Twice

The question nobody is answering

MCP solved how an agent talks to a tool. It did not solve which tools an agent should be holding. As registries grow past a few hundred servers, that second question becomes the expensive one: attach everything and you drown the model in options it will misuse; attach too little and the task is unsolvable before it starts.

Most systems answer it with a retrieval step and leave it there — no measurement, no baseline, no idea what the third pick costs. We wanted a number, on data we did not author.

The benchmark

MCP-Bench (Accenture, Apache-2.0) is built for exactly this. Every task ships a fuzzy description that deliberately names no tool and no server — “I'm prepping for a reactor startup tomorrow and my boss handed me fourteen different readings” — plus the gold server that can solve it and ten distractors. That is a labelled retrieval problem with an honest difficulty setting: the query never tells you the answer.

We used 104 tasks: 56 needing one server, 30 needing two, 18 needing three.

Half one: can we rank the right server first?

This half is deterministic and free — no agent, no rollouts, just ranking. We ran it in two regimes, because a benchmark's own distractor sample can flatter you: against the ten distractors it provides, and against all 28 servers at once, which is closer to a deployment.

Ranking against every server in the registry, the right one comes first 89.4% of the time against a 3.6% chance baseline.

What that headline number hides

any@1 asks whether at least one gold server ranks first. For the 56 tasks that need exactly one server, that is the whole question. For the other 48 — 30 needing two servers, 18 needing three — the question a user actually cares about is whether the retriever assembled the full set.

It often does not. Measuring “all gold servers within the top 5” on the dense arm, against all 28 servers:

The three-server row is the striking one. Dense finds a relevant server on every single one of those tasks — 100% any@1 — and still assembles the complete set under half the time. Finding one relevant server is easy; finding all three is a different problem, and an aggregate any@1 headline hides the gap entirely. Composite tasks are the majority here: 48 of the 104.

The finding that cost us a default

Look at the middle row. Our shipped default was hybrid — reciprocal rank fusion of the lexical and semantic arms, which is close to folk wisdom in retrieval. It lost to plain dense by about ten points in both regimes.

The mechanism is unglamorous: RRF fuses a strong arm with a weak one and lands between them. When your two arms are genuinely complementary that is a win. When one is simply better on this query shape, fusion is a tax.

With one guard. “Dense-only” plus “no embedding endpoint” is an empty result, not a degraded one — every candidate silently dropped, at exactly the moment your embeddings provider is down. Four of our own tests caught this within seconds of the change. So the weights now express a preference and a fallback expresses the guarantee: if the semantic arm returns nothing and the lexical arm has candidates, use them and say so in the log.

Half two: does the agent actually use the right one?

Retrieval quality is not the claim a user cares about. They care whether the agent, handed a pile of servers, calls the right one. So we installed twelve MCP-Bench servers and ran them for real — subprocesses over stdio, genuine JSON-RPC, 103 tools on offer.

Two conditions per task, because “it used the right server” means nothing without a counterfactual. gold-only attaches just the correct server and establishes the task is doable at all. all-12 attaches everything and forces a choice. Scoring is deterministic — did a tool belonging to the gold server get invoked — so there is no judge and no rubric to argue with.

91.3% correct selection against an 8.3% random-pick baseline. But the number we find most interesting is a zero: across 62 tool calls with 103 tools available from twelve servers, the agent called a tool from a wrong server zero times.

Both misses are abstentions — the agent made no tool call at all and answered from its own knowledge — rather than misroutes. Distraction costs coverage here, never correctness. Of the two failure modes, that is the one you would choose.

What we had to fix to measure anything

An awkward discovery: our stdio MCP client was a stub. Every MCP capability we had was running against simulated responses. Making it real surfaced four protocol details, and every one of them fails silently:

• notifications/initialized is required after the handshake. Without it, Python SDK servers answer initialize and then serve an empty tool list — which reads as “this server has no tools” rather than “you skipped a step”. Three of our first four servers did exactly this.
• The subprocess must stay alive between calls. Closing stdin per call makes the server exit mid-reply, so you lose every response after the first.
• Servers print non-protocol lines to stdout. Parse one as a message and every later request/response pairing desynchronises.
• stderr must be drained, or a chatty server fills its pipe buffer and deadlocks.

Two more bugs in the harness itself both produced a clean-looking 0%: the agent was attached to four live servers but shown none of their tools, and our trace parser read the wrong field. In the first case the model responded by inventing a tool that does not exist. Neither crashed. Both would have been reported as “the agent doesn't use MCP tools”.

What we are not claiming

This is a component evaluation using MCP-Bench's task and label data. It is not a run of their end-to-end harness, which measures full agent completion with live tool schemas and scores dimensions we did not touch. We are not reporting a leaderboard position.

MCP-Bench ships launch commands rather than server descriptions, so the registry metadata a real deployment would have ingested was reconstructed by us — from server names only, with no task text in scope. Real registry metadata would be strictly better, and we would expect the numbers to move.

The live half is 24 tasks. 91.3% carries a confidence interval of [73%, 98%] — the direction is solid and the zero-misroute result is clean, but the point estimate is not precise. An earlier four-server run gave 75%, and the intervals overlap: that was small-sample noise, which is exactly why we tripled the sample rather than publishing the first number we liked.

Why this is an axis, not a feature

Retrieval picks a set once, before anything runs, and never revisits it. If the third pick is a distractor, every workflow carries it and nothing ever measures the cost.

So in our optimizer, MCP membership is searchable: one on/off dimension per candidate server, and the attribution reports which servers earn their place. “ This server carries 40% of the variance, the other four are noise” is something you can act on. “Retrieval attached five servers” is not.