Two in 25 Tool Calls Fail: The Quiet Metric Behind Agent Automation
Two in 25 Tool Calls Fail: The Quiet Metric Behind Agent Automation
When an AI agent breaks, it rarely crashes. It misfires quietly: a tool call comes back malformed, the harness returns an error, the error lands in the model's context, and the agent keeps working from a slightly corrupted picture of the world. Multiply that across a multi-step business workflow (routing tickets, reconciling invoices, chasing renewals) and "mostly correct" becomes an operations problem nobody has a dashboard for.
A recent write-up from the team at Ubicloud put a number on it: serving the Kimi K2.6 model on a popular open-source inference engine, only 92% of tool calls matched the expected schema. As they put it, 2 out of every 25 attempted tool calls failed. The agent did not stop. It just got worse.
What a tool call actually is
An agent acts through tools: fetch a page, query a database, create a ticket. The harness tells the model which tools exist and, crucially, hands over a schema: a machine-readable contract describing each tool's parameters, their types, and which ones are required. The model is expected to answer with a structured call that obeys that contract.
The catch is that a language model is a next-token predictor, not a type checker. As the Ubicloud engineers explain, even with a clear schema in front of it, a model can hallucinate a field name it saw during training (search_web instead of web_search, city_name instead of location), or emit JSON that simply doesn't parse. Nothing announces the failure. The harness just gets back something it cannot run.
Where "mostly works" costs you
A malformed tool call triggers a small cascade:
- The harness returns an error message, which enters the model's context window.
- The agent retries, burning tokens, and possibly calling the wrong tool with new confidence.
- In multi-turn agentic workloads, the polluted context degrades every subsequent step.
None of this shows up as an incident. It shows up as higher inference bills, longer runs, and automations that "usually work": the worst possible reliability profile for a business process.
Model providers know this. Moonshot AI publishes the K2VV ToolCall benchmark, a 2,000-request workload that scores an inference provider on trigger similarity (does it call tools at the right moments?) and schema accuracy (are the calls actually executable?). It exists precisely because serving the same model badly can silently degrade it.
The fix lives below the prompt
The interesting part of the Ubicloud story is where the solution sits: not in better prompting, not in a smarter model, but in the inference engine. vLLM, the open-source serving layer, added support for constrained decoding, a technique where the engine guides generation token by token, only allowing tokens that keep the output valid against the tool's schema. It is implemented using structural tags from the XGrammar library, and since vLLM 0.31 it is a server-side flag. With it enabled, Ubicloud reports 100% schema accuracy for the newer Kimi and DeepSeek models: the malformed call becomes structurally impossible rather than merely unlikely.
The takeaway
If your agent stack runs on self-hosted or open-weight models, schema accuracy is a deployment setting, not a model property. The same weights can be 92% or 100% reliable depending on how they are served.
For teams buying agent platforms rather than running them, the same question applies one layer up: ask your provider how they measure tool-call accuracy, and whether constrained decoding is on. A vendor discussion on Hacker News, where the Ubicloud authors answered questions directly, noted that Moonshot AI itself recommends constrained decoding on by default and serves Kimi that way. If the model's creator treats it as the default, "we didn't enable it" is a red flag, not a neutral choice.
Know the limits
Constrained decoding guarantees syntax, not sense. A tool call that validates against the schema can still fetch the wrong URL or query the wrong table. And because the constraint forces the model away from its natural token distribution, practitioners in the Hacker News thread rightly note that overall output quality should be observed with the feature on and off; the Ubicloud authors themselves say mileage varies by model. Treat it as one layer of reliability, not a substitute for verification, approval gates, and logs.
What to actually do
If agents are moving real work through your business, the checklist is short:
- Ask your inference provider or platform vendor for their tool-call schema accuracy numbers. If they don't measure it, that is the answer.
- If you self-host open-weight models, check whether constrained decoding is available in your serving layer and enabled for models that recommend it.
- Watch for the symptoms upstream of the metric: retry storms, growing context sizes, runs that succeed but take longer than they should.
- Keep the human gates for what syntax checking cannot cover: irreversible actions, payments, deletes, anything that leaves the building.
Reliable automation is not one heroic component. It is a stack of small guarantees: schemas enforced at generation time, verification before commit, humans at the irreversible steps. Tool-call accuracy is one of the cheapest guarantees to buy, and the one most teams have never heard of.
If you are working out where guarantees like this belong in your own workflows: talk to us.
