Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer

TL;DR

This paper introduces CRDT, a novel decision transformer framework that uses counterfactual reasoning to improve decision-making and generalization in reinforcement learning, especially with limited or suboptimal data.

Contribution

CRDT is the first to incorporate counterfactual reasoning into decision transformers, enabling reasoning beyond known data without architectural changes.

Findings

CRDT outperforms traditional decision transformers on Atari and D4RL benchmarks.

CRDT demonstrates improved performance with limited and altered data scenarios.

CRDT enables stitching of suboptimal trajectories through counterfactual reasoning.

Abstract

Decision Transformers (DT) play a crucial role in modern reinforcement learning, leveraging offline datasets to achieve impressive results across various domains. However, DT requires high-quality, comprehensive data to perform optimally. In real-world applications, the lack of training data and the scarcity of optimal behaviours make training on offline datasets challenging, as suboptimal data can hinder performance. To address this, we propose the Counterfactual Reasoning Decision Transformer (CRDT), a novel framework inspired by counterfactual reasoning. CRDT enhances DT ability to reason beyond known data by generating and utilizing counterfactual experiences, enabling improved decision-making in unseen scenarios. Experiments across Atari and D4RL benchmarks, including scenarios with limited data and altered dynamics, demonstrate that CRDT outperforms conventional DT approaches. Additionally, reasoning counterfactually allows the DT agent to obtain stitching abilities, combining suboptimal trajectories, without architectural modifications. These results highlight the potential of counterfactual reasoning to enhance reinforcement learning agents' performance and generalization capabilities.