Learning Steerable Clarification Policies with Collaborative Self-play

TL;DR

This paper introduces a method for training AI assistant policies to manage ambiguous queries using self-play and reinforcement learning, enabling context-dependent and cost-sensitive clarification strategies.

Contribution

It presents a novel self-play training approach for steerable clarification policies that adapt based on cost inputs, improving response accuracy and flexibility.

Findings

Steerable policies improve accuracy in ambiguous query handling.

The approach generalizes to unseen cost values.

Reinforced Self-Training enhances policy performance.

Abstract

To handle underspecified or ambiguous queries, AI assistants need a policy for managing their uncertainty to determine (a) when to guess the user intent and answer directly, (b) when to enumerate and answer multiple possible intents, and (c) when to ask a clarifying question. However, such policies are contextually dependent on factors such as user preferences or modality. For example, enumerating multiple possible user intentions is cumbersome on small screens or in a voice setting. In this work, we propose to train steerable policies for managing this uncertainty using self-play. Given two agents, one simulating a user and the other an AI assistant, we generate conversations where the user issues a potentially ambiguous query, and the assistant needs to determine how to respond. Importantly, the model takes as input the numerical cost of each clarification question, and each generated word, and is asked to take the action that will maximize its final reward, which is the cost-penalized accuracy. We use Reinforced Self-Training (ReST) to train our model to achieve high reward and show this leads to a steerable policy that changes its behavior predictably conditioned on the provided costs, leading to higher reward and accuracy. Moreover, our procedure also generalizes to numerical cost values that were unobserved at training time.