Double Check My Desired Return: Transformer with Target Alignment for Offline Reinforcement Learning

TL;DR

This paper introduces Doctor, a transformer-based offline RL method that improves alignment between desired and achieved returns by combining supervised learning and value estimation with a double-check mechanism, enhancing control precision.

Contribution

Doctor is a novel offline RL approach that jointly optimizes action prediction and value estimation, with a double-check inference mechanism for better return alignment.

Findings

Doctor achieves stronger performance on D4RL benchmarks.

It provides more accurate control aligned with target returns.

Demonstrates effectiveness across diverse tasks.

Abstract

Offline reinforcement learning (RL) has achieved significant advances in domains such as robotic control, autonomous driving, and medical decision-making. Most existing methods primarily focus on training policies that maximize cumulative returns from a given dataset. However, many real-world applications require precise control over policy performance levels, rather than simply pursuing the best possible return. Reinforcement learning via supervised learning (RvS) frames offline RL as a sequence modeling task, enabling the extraction of diverse policies by conditioning on different desired returns. Yet, existing RvS-based transformers, such as Decision Transformer (DT), struggle to reliably align the actual achieved returns with specified target returns, especially when interpolating within underrepresented returns or extrapolating beyond the dataset. To address this limitation, we propose Doctor, a novel approach that Double Checks the Transformer with target alignment for Offline RL. Doctor integrates the strengths of supervised learning (SL) and temporal difference (TD) learning by jointly optimizing the action prediction and value estimation. During inference, Doctor introduces a double-check mechanism: actions are first sampled around the desired target returns and then validated with value functions. This ensures more accurate alignment between predicted actions and desired target returns. We evaluate Doctor on the D4RL and EpiCare benchmarks, demonstrating aligned control yields stronger performance and tunable expertise, showing its effectiveness in a wide range of tasks.