TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate

TL;DR

Terms-Bench introduces a Bayesian-game framework for diagnosing large language model negotiation agents, enabling detailed failure analysis beyond simple deal rate metrics.

Contribution

It provides a novel environment that makes the negotiation process itself the verifier, allowing for diagnostic insights into agent failures and strengths.

Findings

Frontier models saturate deal rate but differ in surplus extraction and cue use.

Agents show divergence in belief calibration and compliance.

Prior benchmarks masked agent-specific bargaining bottlenecks.

Abstract

Negotiation is a central mechanism of economic exchange, shaping markets, procurement, labor agreements, and resource allocation. It is also a canonical testbed for agentic language models, requiring multi-turn interaction under hidden preferences, strategic communication, and binding constraints. These properties make negotiation hard to evaluate: unlike math or code, it has no intrinsic verifier. Existing LLM negotiation evaluations rely on LLM-vs.-LLM interaction or aggregate outcomes such as deal rate, leaving failures opaque. We introduce Terms-Bench, short for Testbed for Economic Reasoning in Multi-turn Strategy, a Bayesian-game framework that makes the environment itself the verifier by specifying the counterpart's latent type, policy, and payoff structure. We instantiate it in bilateral price negotiation, where the counterpart's private state and simulator policy are hidden from the agent but observable to the evaluator. This turns the counterpart from a black-box opponent into a diagnostic instrument, enabling agent-attributable failure analysis and oracle-reference optimality gaps. Evaluating 13 LLM agents spanning frontier systems from major providers, Terms-Bench turns negotiation evaluation from aggregate ranking into actionable diagnosis: where agents fail, why they fail, and what to strengthen. Empirically, frontier models saturate deal rate yet diverge in surplus extraction, cue use, belief calibration, and compliance, revealing agent-specific bargaining bottlenecks masked by prior benchmarks.