Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch

TL;DR

This paper introduces a novel transfer learning framework for deep reinforcement learning that enables knowledge transfer between tasks with different state and action spaces using embeddings and mutual information maximization.

Contribution

It proposes a new method for transfer learning in RL with arbitrary state-action mismatch, utilizing embeddings and mutual information to transfer knowledge effectively.

Findings

Successful transfer in robotic locomotion tasks with different state-action spaces

Embeddings enriched by mutual information improve transfer quality

Framework handles complex, high-dimensional tasks effectively

Abstract

Deep reinforcement learning (RL) algorithms have achieved great success on a wide variety of sequential decision-making tasks. However, many of these algorithms suffer from high sample complexity when learning from scratch using environmental rewards, due to issues such as credit-assignment and high-variance gradients, among others. Transfer learning, in which knowledge gained on a source task is applied to more efficiently learn a different but related target task, is a promising approach to improve the sample complexity in RL. Prior work has considered using pre-trained teacher policies to enhance the learning of the student policy, albeit with the constraint that the teacher and the student MDPs share the state-space or the action-space. In this paper, we propose a new framework for transfer learning where the teacher and the student can have arbitrarily different state- and action-spaces. To handle this mismatch, we produce embeddings which can systematically extract knowledge from the teacher policy and value networks, and blend it into the student networks. To train the embeddings, we use a task-aligned loss and show that the representations could be enriched further by adding a mutual information loss. Using a set of challenging simulated robotic locomotion tasks involving many-legged centipedes, we demonstrate successful transfer learning in situations when the teacher and student have different state- and action-spaces.