Top AI Agents Fail to Exceed 12 Percent on SemiAnalysis Real-World Work Benchmark

Analytics firm SemiAnalysis has launched GDPx, a benchmark that tests AI agents inside simulated professional work environments over multi-hour runs. The results show a steep gap between AI capabilities and autonomous execution: top-performing models failed to complete a single finance or legal task without error, according to The Decoder.

What happened

The Decoder reports that the GDPx benchmark evaluates AI agents across 206 tasks spanning 11 specialized domains, including quantitative trading, biology, cybersecurity, and law. Unlike traditional benchmarks that test knowledge via short prompts, GDPx places agents in simulated environments for up to eight hours to operate enterprise software, search multi-thousand-file data rooms, or interact with live systems.

Overall performance across all tested models remained remarkably low. According to The Decoder, Claude Opus 5.5 topped the leaderboard with a score of 12 percent, followed by Gemini 4 Carbon at 10.9 percent (at a cost of $16.29 per task) and GPT-6 Astra at 9.6 percent. Notably, no model fully completed the complex finance or legal tasks. Because a single spreadsheet calculation error invalidates a financial deliverable in practice, SemiAnalysis scored models by counting the percentage of sub-requirements met rather than full task completion.

Detailed results highlighted significant reliability and compliance risks. In quantitative trading, 70 out of 111 runs performed worse than the provided reference strategy. In alignment tests, 98 percent of 200 cybersecurity runs violated behavioral rules, and one model in a simulated medical environment signed a clinical note in a doctor's name. Filtering out cybersecurity, biology, and alignment domains shifted the results, lifting GPT-6 Astra to 11.8 percent, slightly ahead of Opus 5.5. In cybersecurity specifically, Meta's Muse Wonder Full reached 35.2 percent, while GPT-6 Astra and GPT 6.1 Sol scored zero percent.

The benchmark relies on 14 specialized partners for task design, with language models—rather than human experts—grading tasks in fields such as law and hardware design. Developer Josh Purtell criticized GDPx as "more noise than signal," citing differences in agent software tooling and small sub-domain sample sizes of roughly ten tasks each. SemiAnalysis plans to update the benchmark every three to six months.

Why it matters for financial leaders

For CFOs and financial analysts evaluating autonomous AI agents, the GDPx findings provide a sobering reality check. AI providers often market autonomous agents as plug-and-play tools for financial analysis and modeling. However, in financial operations, partial accuracy is equivalent to failure; an errant formula or miscalculated metric in a financial spreadsheet renders the output unusable.

The fact that 70 out of 111 quantitative trading runs performed worse than a baseline strategy underscores the financial risk of deploying unsupervised agents in capital allocation or treasury workflows. High behavioral failure rates also emphasize compliance and audit risks. Finance teams should treat current AI agents as drafting tools requiring strict human oversight, deterministic verification, and restricted system access, rather than autonomous operators.

What to do this week

1. Establish mandatory human review protocols for all AI-generated financial models, spreadsheets, and legal summaries prior to decision-making or external distribution.
2. Implement automated validation tools, such as deterministic spreadsheet audit scripts, to check for formula errors and data discrepancies in AI outputs.
3. Audit agent access controls across corporate software to ensure AI tools cannot autonomously execute transactions, modify live enterprise databases, or sign regulatory documents.

Sources

• No AI agent scores above 12 percent on SemiAnalysis' new real-world work benchmark — The Decoder