Benchmarking Agents in Insurance Underwriting Environments

TL;DR

This paper introduces UNDERWRITE, a realistic insurance underwriting benchmark designed with domain experts, revealing critical gaps in current AI models' performance and highlighting the importance of expert-driven evaluation for enterprise readiness.

Contribution

The paper presents UNDERWRITE, a novel, expert-designed benchmark that incorporates real-world complexities like proprietary knowledge and noisy interfaces, addressing limitations of existing benchmarks.

Findings

Significant performance gaps between research models and enterprise needs.

Most accurate models are not the most efficient or reliable.

Hallucination of domain knowledge persists despite tool access.

Abstract

As AI agents integrate into enterprise applications, their evaluation demands benchmarks that reflect the complexity of real-world operations. Instead, existing benchmarks overemphasize open-domains such as code, use narrow accuracy metrics, and lack authentic complexity. We present UNDERWRITE, an expert-first, multi-turn insurance underwriting benchmark designed in close collaboration with domain experts to capture real-world enterprise challenges. UNDERWRITE introduces critical realism factors often absent in current benchmarks: proprietary business knowledge, noisy tool interfaces, and imperfect simulated users requiring careful information gathering. Evaluating 13 frontier models, we uncover significant gaps between research lab performance and enterprise readiness: the most accurate models are not the most efficient, models hallucinate domain knowledge despite tool access, and pass^k results show a 20% drop in performance. The results from UNDERWRITE demonstrate that expert involvement in benchmark design is essential for realistic agent evaluation, common agentic frameworks exhibit brittleness that skews performance reporting, and hallucination detection in specialized domains demands compositional approaches. Our work provides insights for developing benchmarks that better align with enterprise deployment requirements.