Research reveals systematic validity problems in automated benchmarks evaluating agentic AI systems, with audits finding flaws in 7 of 10 popular benchmarks—raising questions about safety certifica...
Academic paper proposes optimal allocation strategy for auditing LLM agent fleets with limited human budget, identifying miscalibration thresholds where confidence-based auditing underperforms rand...
Research on 99,952 examples shows correct domain rubrics improve LLM evaluator accuracy by 2.11 points, suggesting specialization matters more in evaluation rules than model weights.
Researchers introduce FinanceHarness, an agentic AI framework designed specifically for financial deep research that combines LLMs with layered agent architecture and point-in-time benchmarks to au...
arXiv paper introduces ProofAgent Index (PAI), a governance readiness framework for AI agents in production, addressing gap between capability demos and real-world operational constraints in accoun...
Researchers propose Hierarchical Reranker, a RAG framework designed to improve how AI systems retrieve and analyze large-scale financial documents like 10-K filings and tabular disclosures, address...
Mercor and Ramp launch APEX-Accounting, a 160-task benchmark evaluating whether frontier AI models can perform core accounting work like reconciliation, accrual, transaction posting, and reporting.
Mercor and Ramp release APEX-Accounting, a research benchmark with 160 expert-authored tasks assessing whether frontier AI models can perform actual accounting work like reconciliations, accruals, ...
GAUGE benchmark evaluates agent-built financial forecasts using multiple independent analyst models rather than single reference, addressing the challenge that reasonable financial projections admi...
Researcher audits 40 empirical records on generative AI in accounting, finding that model obsolescence and publication lag create systematic validity gaps—proposing a claim-currency framework to as...
Academic research using real estate case studies to help firms distinguish between promising and doomed AI projects, with frameworks for ROI estimation and project selection.
New authorization audit method evaluates how LLM agents select tools and arguments, addressing risks when agents rely on untrusted or unauthorized evidence in accounting and compliance tasks.
Academic paper examines how declared vs. implemented identity rules in record systems can diverge without triggering audit detections, with implications for financial data integrity and audit proce...
New evaluation framework tests AI systems on multilingual financial question answering across English, Chinese, Japanese, Spanish, and Greek with 256 benchmark items.
Mercor and Ramp introduce APEX-Accounting, a benchmark measuring whether AI agents can execute core accounting tasks like reconciliation, variance analysis, and month-end close.
Researcher proposes adapting food safety's HACCP methodology to create an auditable self-control standard for AI, bridging compliance gaps in automated systems used by auditors and accountants.
Researchers distilled DeepSeek V4 Flash into a 120B open-weights model scoring 83.61% on FinanceReasoning benchmark, outperforming Kimi K3 and Inkling, while confirming censorship did not transfer ...
Systima's testing reveals Claude's subagent architecture wastes 2–6x input tokens and offers no speed gains over sequential task processing, raising cost concerns for AI automation workflows.
Kimi K3 evaluated on TaxCalcBench for ability to file American tax returns, testing cross-border AI capability in specialized tax domain.
External audits rely heavily on manual testing despite automation advances, creating efficiency and risk gaps as accounting systems become increasingly digital.