Dual Alignment Maximin Optimization for Offline Model-based RL

TL;DR

This paper introduces DAMO, a novel offline RL framework that aligns policies with models and data, improving consistency and performance in distribution mismatch scenarios.

Contribution

The paper proposes a new actor-critic paradigm, Dual Alignment Maximin Optimization, focusing on policy discrepancies and ensuring model-policy consistency in offline RL.

Findings

DAMO achieves competitive results across benchmark tasks.

It effectively aligns policies with models and data.

The framework improves robustness against distribution mismatch.

Abstract

Offline reinforcement learning agents face significant deployment challenges due to the synthetic-to-real distribution mismatch. While most prior research has focused on improving the fidelity of synthetic sampling and incorporating off-policy mechanisms, the directly integrated paradigm often fails to ensure consistent policy behavior in biased models and underlying environmental dynamics, which inherently arise from discrepancies between behavior and learning policies. In this paper, we first shift the focus from model reliability to policy discrepancies while optimizing for expected returns, and then self-consistently incorporate synthetic data, deriving a novel actor-critic paradigm, Dual Alignment Maximin Optimization (DAMO). It is a unified framework to ensure both model-environment policy consistency and synthetic and offline data compatibility. The inner minimization performs dual conservative value estimation, aligning policies and trajectories to avoid out-of-distribution states and actions, while the outer maximization ensures that policy improvements remain consistent with inner value estimates. Empirical evaluations demonstrate that DAMO effectively ensures model and policy alignments, achieving competitive performance across diverse benchmark tasks.