SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning

TL;DR

SpeakRL is a reinforcement learning approach that improves language models' ability to proactively engage in conversations by asking clarifying questions, leading to better task completion without increasing dialogue length.

Contribution

The paper introduces SpeakRL, a novel RL method for training agents to proactively clarify user intents, and curates SpeakER, a synthetic dataset for training and evaluation.

Findings

20.14% improvement in task completion

Effective reward design for conversational proactivity

Proactive clarification enhances user-agent collaboration

Abstract

Effective human-agent collaboration is increasingly prevalent in real-world applications. Current trends in such collaborations are predominantly unidirectional, with users providing instructions or posing questions to agents, where agents respond directly without seeking necessary clarifications or confirmations. However, the evolving capabilities of these agents require more proactive engagement, where agents should dynamically participate in conversations to clarify user intents, resolve ambiguities, and adapt to changing circumstances. Existing prior work under-utilize the conversational capabilities of language models (LMs), thereby optimizing agents as better followers rather than effective speakers. In this work, we introduce SpeakRL, a reinforcement learning (RL) method that enhances agents' conversational capabilities by rewarding proactive interactions with users, such as asking right clarification questions when necessary. To support this, we curate SpeakER, a synthetic dataset that includes diverse scenarios from task-oriented dialogues, where tasks are resolved through interactive clarification questions. We present a systematic analysis of reward design for conversational proactivity and propose a principled reward formulation for teaching agents to balance asking with acting. Empirical evaluations demonstrate that our approach achieves a 20.14% absolute improvement in task completion over base models without increasing conversation turns even surpassing even much larger proprietary models, demonstrating the promise of clarification-centric user-agent interactions.