Online Finetuning Decision Transformers with Pure RL Gradients

TL;DR

This paper introduces novel algorithms for online finetuning of Decision Transformers using pure reinforcement learning gradients, overcoming previous limitations and achieving state-of-the-art results in various benchmarks.

Contribution

It presents the first methods enabling online finetuning of Decision Transformers with pure RL gradients, including adaptations of GRPO and new techniques for stability and exploration.

Findings

Outperforms existing online Decision Transformer methods.

Achieves state-of-the-art results across multiple benchmarks.

Demonstrates stability and efficiency improvements in online RL finetuning.

Abstract

Decision Transformers (DTs) have emerged as a powerful framework for sequential decision making by formulating offline reinforcement learning (RL) as a sequence modeling problem. However, extending DTs to online settings with pure RL gradients remains largely unexplored, as existing approaches continue to rely heavily on supervised sequence-modeling objectives during online finetuning. We identify hindsight return relabeling -- a standard component in online DTs -- as a critical obstacle to RL-based finetuning: while beneficial for supervised learning, it is fundamentally incompatible with importance sampling-based RL algorithms such as GRPO, leading to unstable training. Building on this insight, we propose new algorithms that enable online finetuning of Decision Transformers using pure reinforcement learning gradients. We adapt GRPO to DTs and introduce several key modifications, including sub-trajectory optimization for improved credit assignment, sequence-level likelihood objectives for enhanced stability and efficiency, and active sampling to encourage exploration in uncertain regions. Through extensive experiments, we demonstrate that our methods outperform existing online DT baselines and achieve new state-of-the-art performance across multiple benchmarks, highlighting the effectiveness of pure-RL-based online finetuning for Decision Transformers.