LHAW: Controllable Underspecification for Long-Horizon Tasks

TL;DR

LHAW introduces a modular pipeline to systematically create and evaluate task variants with controlled ambiguity, enabling better assessment and development of autonomous agents for long-horizon tasks.

Contribution

The paper presents LHAW, a novel framework for generating and measuring the impact of ambiguity in long-horizon tasks, facilitating scalable, task-agnostic evaluation of agent clarification capabilities.

Findings

285 task variants released for evaluation

Current agents show varied ability to detect and resolve underspecification

LHAW enables cost-sensitive assessment of clarification strategies

Abstract

Long-horizon workflow agents that operate effectively over extended periods are essential for truly autonomous systems. Their reliable execution critically depends on the ability to reason through ambiguous situations in which clarification seeking is necessary to ensure correct task execution. However, progress is limited by the lack of scalable, task-agnostic frameworks for systematically curating and measuring the impact of ambiguity across custom workflows. We address this gap by introducing LHAW (Long-Horizon Augmented Workflows), a modular, dataset-agnostic synthetic pipeline that transforms any well-specified task into controllable underspecified variants by systematically removing information across four dimensions - Goals, Constraints, Inputs, and Context - at configurable severity levels. Unlike approaches that rely on LLM predictions of ambiguity, LHAW validates variants through empirical agent trials, classifying them as outcome-critical, divergent, or benign based on observed terminal state divergence. We release 285 task variants from TheAgentCompany, SWE-Bench Pro and MCP-Atlas according to our taxonomy alongside formal analysis measuring how current agents detect, reason about, and resolve underspecification across ambiguous settings. LHAW provides the first systematic framework for cost-sensitive evaluation of agent clarification behavior in long-horizon settings, enabling development of reliable autonomous systems.