Transfer learning with causal counterfactual reasoning in Decision Transformers

TL;DR

This paper introduces a method for transfer learning in reinforcement learning by applying causal counterfactual reasoning within the Decision Transformer framework, enabling adaptation to new environments while maintaining performance.

Contribution

It presents a novel approach combining causal reasoning with Decision Transformers for effective transfer learning in changing environments.

Findings

Successful transfer of policies with minimal reward loss

Effective use of factual and counterfactual data for policy distillation

Enhanced adaptability in reinforcement learning environments

Abstract

The ability to adapt to changes in environmental contingencies is an important challenge in reinforcement learning. Indeed, transferring previously acquired knowledge to environments with unseen structural properties can greatly enhance the flexibility and efficiency by which novel optimal policies may be constructed. In this work, we study the problem of transfer learning under changes in the environment dynamics. In this study, we apply causal reasoning in the offline reinforcement learning setting to transfer a learned policy to new environments. Specifically, we use the Decision Transformer (DT) architecture to distill a new policy on the new environment. The DT is trained on data collected by performing policy rollouts on factual and counterfactual simulations from the source environment. We show that this mechanism can bootstrap a successful policy on the target environment while retaining most of the reward.