A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithms

TL;DR

This paper compares Decision Transformers and traditional offline RL algorithms, analyzing their performance across different reward structures and data qualities in the ANT environment, highlighting their respective strengths and limitations.

Contribution

It provides a comprehensive empirical comparison of Decision Transformers and traditional offline RL methods in various reward and data quality settings.

Findings

DTs are less sensitive to reward density variations.

DTs excel in sparse reward scenarios with medium-expert data.

Traditional methods perform better in dense reward environments.

Abstract

The field of Offline Reinforcement Learning (RL) aims to derive effective policies from pre-collected datasets without active environment interaction. While traditional offline RL algorithms like Conservative Q-Learning (CQL) and Implicit Q-Learning (IQL) have shown promise, they often face challenges in balancing exploration and exploitation, especially in environments with varying reward densities. The recently proposed Decision Transformer (DT) approach, which reframes offline RL as a sequence modelling problem, has demonstrated impressive results across various benchmarks. This paper presents a comparative study evaluating the performance of DT against traditional offline RL algorithms in dense and sparse reward settings for the ANT continous control environment. Our research investigates how these algorithms perform when faced with different reward structures, examining their ability to learn effective policies and generalize across varying levels of feedback. Through empirical analysis in the ANT environment, we found that DTs showed less sensitivity to varying reward density compared to other methods and particularly excelled with medium-expert datasets in sparse reward scenarios. In contrast, traditional value-based methods like IQL showed improved performance in dense reward settings with high-quality data, while CQL offered balanced performance across different data qualities. Additionally, DTs exhibited lower variance in performance but required significantly more computational resources compared to traditional approaches. These findings suggest that sequence modelling approaches may be more suitable for scenarios with uncertain reward structures or mixed-quality data, while value-based methods remain competitive in settings with dense rewards and high-quality demonstrations.