The Judgment Gap: Why the Best AI Agent Still Fails Nine in Ten Professional Assignments
The Judgment Gap: Why the Best AI Agent Still Fails Nine in Ten Professional Assignments
If your automation roadmap assumes an AI agent can take a messy client engagement from "here are the files" to "here is the finished work," a benchmark published this week is worth ten minutes of your time. Surge AI released DAYJOB: Tax on October 27: 50 professional tax assignments, real client records, and a pass bar set at professional standards. The best model on the leaderboard passed 10.8% of attempts.
The interesting part is not the small percentage. It is where the failures come from. Models mostly got the calculations right. What they missed was the part of the job you actually hire a professional for.
The benchmark: real client folders, not exam questions
Most AI benchmarks start with a defined problem. Answer this legal question. Fill in this missing figure. DAYJOB: Tax starts the way a real engagement starts: a short request from a client, a folder of records, and no list of what matters.
The numbers behind the benchmark (from Surge AI's announcement and the dataset page on Hugging Face):
- 50 assignments across eight areas of tax practice, from individual returns to partnerships, estates, international tax, and disputes.
- 1,789 workspace files, with a median of 30 per assignment and as many as 123. That includes 1,487 PDFs, 117 Excel workbooks, and 38 Word documents, plus emails, images, and CSVs.
- Messy evidence by design: a handwritten receipt photographed from a desk, a field dispatch log showing a company works in five states when the client assumed one, cross-border records that have to be reconciled line by line.
- Expert grading: tax professionals with backgrounds at firms like PwC, Deloitte, and EY built the tasks, and each one passed a three-layer expert review for correctness, completeness, and solvability.
An attempt passes only when it satisfies at least 95% of the task's rubric criteria, and the published metric is pass@1, a single attempt with no best-of-five retries. The rubric checks calculations, supporting evidence, consequential issues, and the final deliverable.
The scoreboard
Claude Sonnet 5.5 leads at 10.8%, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. Below the top three, scores drop fast: GPT-6 Astra passes 4.8%, Gemini 4 Argon 3.2%, and several well-known models pass nothing at all (the Fireworks Specialized Intelligence Index lists all 16 evaluated models, last updated October 28).
Surge's own summary line is the honest takeaway: "Agents haven't yet passed the audit."
One caveat worth stating before anyone quotes the number in a board deck: no human baseline appears in the published results. The 10.8% is not an AI-versus-tax-professional score. It measures models against an expert rubric, nothing more.
Where agents fail: the question the client never asked
The two failure cases Surge describes are worth reading closely, because neither is a math error.
In the first, a client named Anna planned to sell her condo and asked whether closing in October 2026 or January 2027 was better for taxes. The agent compared the dates correctly: either one satisfied the residence requirement for a tax exclusion worth up to $250,000 in gains. What it never flagged was the deadline. Under the usual rules, homeowners must have lived in the property for at least two of the five years before selling. Anna had moved out in August 2024, so her eligibility would lapse around the end of August 2027. If the sale slipped past that date, she could lose the exclusion entirely. The model Astra missed that deadline in five out of five attempts; Opus caught it in all five.
In the second, a separating couple asked how to divide a Roth IRA and a 401(k). Opus 5.5 opened with "You aren't legally married," and recommended keeping the accounts separate. The couple's own records raised doubts: they had previously lived in Texas, which recognizes informal marriages, and held a ceremony in Colorado. That question, whether a valid marriage exists, changes the tax consequences of everything else. Across all five attempts, Opus built its recommendations on the unmarried assumption anyway. Astra thought to verify marital status in four of five.
Surge's framing is the one that matters for businesses: the agents could perform the individual steps, but exercising professional judgment, deciding what deserves investigation when the client does not ask about it, is where the work falls apart.
The cost side of the frontier
There is an economics layer to this too. On the Fireworks index, Claude Sonnet 5.5 costs $10.71 per task while DeepSeek V4.1 Flash, the best open-weight model on the board, costs $0.49 and passes 1.2%. Even the frontier price point buys roughly one professional-grade deliverable in ten.
That combination, high cost and low pass rate, is not a reason to write agents off. It is a signal about which slices of the job they are ready for today.
What this means for your automation plans
If you are planning agent deployments in any professional-services-adjacent domain (accounting, legal, compliance, finance, insurance), the benchmark suggests a specific division of labor:
- Scope agents to the defined parts. Answering a well-specified question, reconciling a ledger, drafting a first-pass workpaper. This is where current models are strong.
- Keep ownership of "what else should we look at" with a human. The consequential-issue finding is exactly the failure mode the benchmark isolates. It is also the part that carries liability.
- Grade pilots like the benchmark does. Define an acceptance rubric before the pilot, require a high threshold (90% or better), and count single attempts. A pilot scored on average usefulness will hide the misses that a client would experience.
- Budget for review, not just tokens. At $10.71 per task and a 10.8% pass rate, the cost of a review gate per deliverable is small relative to the cost of an unchecked miss.
- Do not extrapolate the 10.8%. The rubric is deliberately strict, as it should be for tax work. Lower-stakes, well-specified tasks will sit far higher on the same curve.
None of this is a surprise if you have run an agent pilot in production. What the benchmark adds is a clean, expert-built measurement of the gap between "the agent did the steps" and "the agent did the job."
The takeaway
The frontier moved again this year, and it will move again next year. But the pattern in these results is stable across every model evaluated: calculation is largely solved, professional judgment is not. Automation plans that give agents the defined work and reserve the judgment calls for people will capture the value available today, and they will be the first to benefit as the judgment gap closes.
Building agents into a business process and need a realistic view of what to delegate first? Contact our team and we will help you scope it.
Sources
- DAYJOB: Tax: Evaluating AI Agents on Professional Tax Work, Surge AI (October 27, 2026)
- DAYJOB: Tax benchmark page and leaderboard, Surge AI
- DAYJOB: Tax leaderboard, Fireworks Specialized Intelligence Index (last updated October 28, 2026)
- surgeai/DAYJOB-tax dataset card, Hugging Face
- DAYJOB: Tax coverage, AI/TLDR (October 27, 2026)
- Hacker News discussion (October 28, 2026)
