Automatic Constraint Policy Optimization based on Continuous Constraint Interpolation Framework for Offline Reinforcement Learning

TL;DR

This paper introduces a unified framework for offline RL constraints called Continuous Constraint Interpolation (CCI), enabling smooth transitions between constraint types and improving policy performance through an adaptive primal-dual algorithm.

Contribution

It proposes CCI as a unified principle for various constraint families and develops ACPO, an adaptive algorithm that optimizes constraints for offline RL.

Findings

Achieves state-of-the-art results on D4RL and NeoRL2 benchmarks.

Demonstrates robustness and versatility across diverse offline RL domains.

Provides theoretical performance bounds and insights for constraint optimization.

Abstract

Offline Reinforcement Learning (RL) relies on policy constraints to mitigate extrapolation error, where both the constraint form and constraint strength critically shape performance. However, most existing methods commit to a single constraint family: weighted behavior cloning, density regularization, or support constraints, without a unified principle that explains their connections or trade-offs. In this work, we propose Continuous Constraint Interpolation (CCI), a unified optimization framework in which these three constraint families arise as special cases along a common constraint spectrum. The CCI framework introduces a single interpolation parameter that enables smooth transitions and principled combinations across constraint types. Building on CCI, we develop Automatic Constraint Policy Optimization (ACPO), a practical primal--dual algorithm that adapts the interpolation parameter via a Lagrangian dual update. Moreover, we establish a maximum-entropy performance difference lemma and derive performance lower bounds for both the closed-form optimal policy and its parametric projection. Experiments on D4RL and NeoRL2 demonstrate robust gains across diverse domains, achieving state-of-the-art performance overall.