HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?

TL;DR

HiL-Bench introduces a benchmark to evaluate agents' ability to recognize when to ask for help, revealing a significant judgment gap in current models and demonstrating trainability of this skill.

Contribution

The paper presents HiL-Bench, a new benchmark with a core metric for help-seeking, and shows that judgment is trainable, improving help-asking behavior in large models.

Findings

No frontier model exceeds half of full-information performance in help-seeking.

Models exhibit three help-seeking failure patterns: overconfidence, high uncertainty with errors, broad escalation.

RL training improves help-seeking quality and transferability across domains.

Abstract

Frontier coding agents solve complex tasks when given complete context but collapse when specifications are incomplete or ambiguous. The bottleneck is not raw capability, but judgment: knowing when to act autonomously and when to ask for help. Current benchmarks are blind to this failure mode. They supply unambiguous detailed instructions and solely reward execution correctness, so an agent that makes a lucky guess for a missing requirement will score identically to one that would have asked to be certain. We present HiL-Bench (Human-in-the-Loop Benchmark) to measure this selective escalation skill. Each task contains human-validated blockers (missing information, ambiguous requests, contradictory information) that surface only through progressive exploration, not upfront inspection. Our core metric, Ask-F1, the harmonic mean of question precision and blocker recall, captures the tension between over-asking and silent guessing; its structure architecturally prevents gaming through question spam. Evaluation across SWE and text-to-SQL domains reveals a large universal judgment gap: no frontier model recovers more than a fraction of its full-information performance when deciding whether to ask. Failure analysis identifies three key help-seeking patterns: overconfident wrong beliefs with no gap detection; high uncertainty detection yet persistent errors; broad, imprecise escalation without self-correction. These consistent patterns confirm poor help-seeking is a model-level flaw, not task-specific. RL training on shaped Ask-F1 reward shows judgment is trainable: a 32B model improves both help-seeking quality and task pass rate, with gains that transfer across domains. The model does not learn domain-specific heuristics for when to ask; it learns to detect unresolvable uncertainty and act on it.