The largest public benchmark of agents using real MCP servers put the best model at 78.95%. That is Claude Sonnet 4 on LiveMCPBench, across 95 tasks spanning 70 servers and 527 tools. Claude Opus 4 came second at 70.53%. Most of the other models tested landed between 30% and 50%.
The bottom of that range is the striking part: on real servers, doing real tasks, a majority of the models tested fail more often than they succeed. And the paper is clear about where the failures concentrate — not in reasoning, but in retrieval, with agents failing to find the semantically right tool in a large catalogue. The authors describe those retrieval errors as dominating the error distribution.
That is not a model capability story. That is a story about the surfaces those models were pointed at.
So the obvious question for anyone running an MCP server is: what is my number? And here the tooling gets strangely quiet.
The category has agreed to measure something else
Go looking for guidance on measuring an MCP server and you will find a consistent recommendation: track call volume, success and failure rates, tool usage, active users, client distribution, latency. All useful. None of it is task completion.
Where task completion does come up, it is usually to explain why it isn't on the list. The common position is that it is the ideal metric but not feasible to track directly, so you should fall back on proxies — typically cost and latency, on the reasoning that fewer tokens and faster responses correlate with the agent getting there.
The proxies are backwards, and dangerously so. Token consumption falls when an agent gives up early. A session where the agent made two calls, hit a schema error it couldn't recover from, and abandoned the task looks cheaper and faster than one where it worked patiently through six calls and succeeded. Optimise on cost and latency alone and you will be rewarded for failing quickly.
The reason everyone reaches for proxies is a real one: your server cannot see the outcome. You get a tool call. You do not get the user's satisfaction, the assistant's summary, or a completion event at the end. There is no callback that says the human got what they wanted.
But "I cannot observe the outcome directly" is not the same as "I cannot measure it". Product analytics has never observed intent either. It infers.
What the session shape tells you
A task that succeeded and a task that was abandoned leave different traces, and the difference is legible without ever seeing the user.
Terminal action reached. Most task types have a call that only happens when the job is essentially done — the write, the submission, the purchase, the state change. Sessions that reach it are a floor on completion, not the whole picture, but a hard floor you can trust.
Abandonment shape. A session that stops immediately after an error, with no recovery attempt, is an abandoned task. This is not speculative: LiveMCPBench documents agents abandoning tasks after failures without attempting retries or alternatives, and it is one of the most reliable signals your traces carry.
Diversion. The agent fails on one tool and its next call is to a different, adjacent tool. It is trying to route around your contract. Diversion is a stronger failure signal than the error itself, because it tells you the agent judged the tool unusable rather than the input wrong.
Retry loops. Repeated calls to the same tool with mutated arguments is an agent trying to guess a contract you failed to document. Sometimes it guesses right, which is worse, because the task completes and the underlying defect stays invisible.
First-try success rate. Of the calls to a given tool, the share that succeed without a retry. This is the leading indicator: it moves before completion does, it is cheap to compute, and it localises the problem to a specific tool instead of a general malaise.
None of these require a callback. All of them are in the traffic you already have, if anything is recording sessions rather than individual requests.
Measure it per tool, not per server
A server-level completion rate is a number you can put on a slide and do nothing with. The unit of work is the tool, because the tool is what you change.
This matters more than it sounds. Averages hide exactly the thing you are looking for: one badly-typed parameter on one tool, stalling every task that touches it, while eleven other tools work fine and hold the average up. Teng Li's scan of 36 popular MCP servers found roughly a third failing their agents, and noted that in almost every failing case the descriptions score was zero while other metrics looked acceptable. The damage was concentrated and specific. A server-wide average would have shown a mild dip.
The two things that make the number trustworthy
Version the contract. "Before" and "after" have to mean which version of the description and schema the agent actually saw, not a timestamp. Descriptions are a production API surface — they change routing behaviour as surely as code does — so they need versioning like code.
Compare against a baseline, not against last month. Agent traffic is not stationary. A frontier model release can shift how agents use your server more than your own change did, in the same week, in the same direction. Read a raw time series after shipping a fix and you will take credit for someone else's model upgrade. That is not a hypothetical risk; it is the default outcome of measuring naively in a fast-moving ecosystem.
And ship changes one at a time where you can. Four contract changes in a week gets you a completion number that went up and no idea which change earned it — which means next quarter you are guessing again.
The short version
The ceiling on the benchmark is not the models. It is the surfaces they are pointed at, and the surfaces are measurable. Task completion cannot be observed directly, so infer it: terminal actions reached, abandonment after error, diversion to adjacent tools, retry loops, and first-try success per tool. Version your contracts so before-and-after means something, and hold the population steady so you are measuring your work rather than Anthropic's.
That inference is what Vesta is built to do — read agent sessions, find where completion is leaking, name the specific change to the specific tool, and check whether it moved. If you want to know what your number is, request access.
Sources
- LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? — Mo Guozhao et al. (95 tasks, 70 servers, 527 tools; Claude Sonnet 4 at 78.95%, Opus 4 at 70.53%, most models 30–50%; retrieve errors dominating the error distribution; agents abandoning tasks without retries)
- I lint-scanned 36 popular MCP servers. A third of them are failing your agent. — Teng Li, July 2026 (a third graded D/F; description scores of zero on almost every failing server)
- MCP Server Analytics: Key Metrics (the conventional metric set: call volume, success/failure rates, active users, client distribution)
- Running Efficient MCP Servers in Production (task completion treated as ideal but impractical; cost and latency proposed as proxies)
If you build a surface that agents use, and you want to know how they actually use it, Vesta is in early access.