Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems

TL;DR

This paper introduces a scalable constrained policy optimization framework for conversational AI that balances user satisfaction improvements with domain-specific safety constraints, using a novel meta-gradient approach.

Contribution

It proposes a new meta-gradient learning method for adaptive constraint satisfaction in domain-specific conversational AI policy optimization.

Findings

Achieves a better balance between policy value and constraint satisfaction.

Demonstrates effectiveness on real-world conversational AI data.

Outperforms existing methods in constraint adherence and user satisfaction.

Abstract

Recently, self-learning methods based on user satisfaction metrics and contextual bandits have shown promising results to enable consistent improvements in conversational AI systems. However, directly targeting such metrics by off-policy bandit learning objectives often increases the risk of making abrupt policy changes that break the current user experience. In this study, we introduce a scalable framework for supporting fine-grained exploration targets for individual domains via user-defined constraints. For example, we may want to ensure fewer policy deviations in business-critical domains such as shopping, while allocating more exploration budget to domains such as music. Furthermore, we present a novel meta-gradient learning approach that is scalable and practical to address this problem. The proposed method adjusts constraint violation penalty terms adaptively through a meta objective that encourages balanced constraint satisfaction across domains. We conduct extensive experiments using data from a real-world conversational AI on a set of realistic constraint benchmarks. Based on the experimental results, we demonstrate that the proposed approach is capable of achieving the best balance between the policy value and constraint satisfaction rate.