When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment

TL;DR

Transformers significantly improve memory in RL by enabling algorithms to recall observations up to 1500 steps ago, but they do not enhance the ability to perform long-term credit assignment, clarifying their role in RL success.

Contribution

The paper introduces formal measures for memory and credit assignment lengths and empirically demonstrates that Transformers improve memory but not credit assignment in RL.

Findings

Transformers extend memory capacity up to 1500 steps.

Transformers do not improve long-term credit assignment.

Empirical measures distinguish memory from credit assignment in RL.

Abstract

Reinforcement learning (RL) algorithms face two distinct challenges: learning effective representations of past and present observations, and determining how actions influence future returns. Both challenges involve modeling long-term dependencies. The Transformer architecture has been very successful to solve problems that involve long-term dependencies, including in the RL domain. However, the underlying reason for the strong performance of Transformer-based RL methods remains unclear: is it because they learn effective memory, or because they perform effective credit assignment? After introducing formal definitions of memory length and credit assignment length, we design simple configurable tasks to measure these distinct quantities. Our empirical results reveal that Transformers can enhance the memory capability of RL algorithms, scaling up to tasks that require memorizing observations $1500$ steps ago. However, Transformers do not improve long-term credit assignment. In summary, our results provide an explanation for the success of Transformers in RL, while also highlighting an important area for future research and benchmark design. Our code is open-sourced at https://github.com/twni2016/Memory-RL