AdaCred: Adaptive Causal Decision Transformers with Feature Crediting

TL;DR

AdaCred introduces a causal graph-based approach in reinforcement learning that emphasizes short-term sequences and adaptive crediting, leading to improved performance and efficiency over traditional long-sequence models.

Contribution

It proposes a novel method that constructs causal graphs from short-term sequences and adaptively credits relevant features, enhancing offline RL and imitation learning.

Findings

Requires shorter trajectory sequences for effective learning

Outperforms conventional methods in offline RL and imitation learning

Retains only task-relevant representations through adaptive pruning

Abstract

Reinforcement learning (RL) can be formulated as a sequence modeling problem, where models predict future actions based on historical state-action-reward sequences. Current approaches typically require long trajectory sequences to model the environment in offline RL settings. However, these models tend to over-rely on memorizing long-term representations, which impairs their ability to effectively attribute importance to trajectories and learned representations based on task-specific relevance. In this work, we introduce AdaCred, a novel approach that represents trajectories as causal graphs built from short-term action-reward-state sequences. Our model adaptively learns control policy by crediting and pruning low-importance representations, retaining only those most relevant for the downstream task. Our experiments demonstrate that AdaCred-based policies require shorter trajectory sequences and consistently outperform conventional methods in both offline reinforcement learning and imitation learning environments.