Dialogue Telemetry: Turn-Level Instrumentation for Autonomous Information Gathering

TL;DR

This paper introduces Dialogue Telemetry, a framework for turn-level monitoring in autonomous information-gathering dialogues, enabling detection of inefficiencies and improving policy performance in simulated search-and-rescue scenarios.

Contribution

We propose Dialogue Telemetry, a novel, model-agnostic measurement framework with signals for residual information and stalling detection, enhancing dialogue monitoring and policy optimization.

Findings

DT effectively distinguishes efficient and stalled dialogues.

Integration of DT signals improves reinforcement learning policy performance.

DT provides interpretable, turn-level instrumentation for dialogue analysis.

Abstract

Autonomous systems conducting schema-grounded information-gathering dialogues face an instrumentation gap, lacking turn-level observables for monitoring acquisition efficiency and detecting when questioning becomes unproductive. We introduce Dialogue Telemetry (DT), a measurement framework that produces two model-agnostic signals after each question-answer exchange: (i) a Progress Estimator (PE) quantifying residual information potential per category (with a bits-based variant), and (ii) a Stalling Index (SI) detecting an observable failure signature characterized by repeated category probing with semantically similar, low-marginal-gain responses. SI flags this pattern without requiring causal diagnosis, supporting monitoring in settings where attributing degradation to specific causes may be impractical. We validate DT in controlled search-and-rescue (SAR)-inspired interviews using large language model (LLM)-based simulations, distinguishing efficient from stalled dialogue traces and illustrating downstream utility by integrating DT signals into a reinforcement learning (RL) policy. Across these settings, DT provides interpretable turn-level instrumentation that improves policy performance when stalling carries operational costs.