ICPO: Illocution-Calibrated Policy Optimization for Multi-Turn Conversation

TL;DR

This paper introduces ICPO, a training framework that improves multi-turn conversational AI by making models more sensitive to ambiguity, leading to more humble and effective interactions.

Contribution

ICPO is a novel training method that incorporates instruction ambiguity and illocutionary intent, enhancing model humility and robustness in multi-turn conversations.

Findings

75% average improvement in multi-turn conversation performance

Models exhibit increased humility and clarification-seeking behavior

Preserves performance on single-turn benchmarks

Abstract

Large Language Models (LLMs) in multi-turn conversations often suffer from a ``lost-in-conversation'' phenomenon, where they struggle to recover from early incorrect assumptions, particularly when users provide ambiguous initial instructions. We find that standard post-training techniques like Reinforcement Learning with Verifiable Rewards (RLVR) exacerbate this issue by rewarding confident, direct answers, thereby inducing overconfidence and discouraging the model from seeking clarification. To address this, we propose Illocution-Calibrated Policy Optimization (ICPO), a novel training framework that sensitizes the model to instruction ambiguity. ICPO augments the training corpus with underspecified prompts and conditions the reward signal on the user's illocutionary intent, rewarding the model for expressing uncertainty or asking for clarification when faced with ambiguity. Experiments demonstrate that ICPO fosters appropriate humility, yielding a substantial average improvement of 75\% in multi-turn conversation, while preserving robust performance on single-turn benchmarks. Our work presents a practical path toward more robust and collaborative conversational AI that can better navigate the nuances of human interaction.