Transfer RL via the Undo Maps Formalism

TL;DR

This paper introduces TvD, a novel transfer reinforcement learning framework that uses distribution matching and optimal transport to adapt policies across environments with different state spaces, without modifying the original policies.

Contribution

The paper proposes a data-centric transfer method using optimal transport to learn environment transformations, enabling policy transfer without domain-specific assumptions.

Findings

Successful transfer across environment transformations in gridworlds

Effective distribution matching via optimal transport

Policy transfer without modifying original policies

Abstract

Transferring knowledge across domains is one of the most fundamental problems in machine learning, but doing so effectively in the context of reinforcement learning remains largely an open problem. Current methods make strong assumptions on the specifics of the task, often lack principled objectives, and -- crucially -- modify individual policies, which might be sub-optimal when the domains differ due to a drift in the state space, i.e., it is intrinsic to the environment and therefore affects every agent interacting with it. To address these drawbacks, we propose TvD: transfer via distribution matching, a framework to transfer knowledge across interactive domains. We approach the problem from a data-centric perspective, characterizing the discrepancy in environments by means of (potentially complex) transformation between their state spaces, and thus posing the problem of transfer as learning to undo this transformation. To accomplish this, we introduce a novel optimization objective based on an optimal transport distance between two distributions over trajectories -- those generated by an already-learned policy in the source domain and a learnable pushforward policy in the target domain. We show this objective leads to a policy update scheme reminiscent of imitation learning, and derive an efficient algorithm to implement it. Our experiments in simple gridworlds show that this method yields successful transfer learning across a wide range of environment transformations.