Component Transfer Learning for Deep RL Based on Abstract Representations

TL;DR

This paper proposes a transfer learning method for deep reinforcement learning that leverages shared low-dimensional abstract representations to transfer knowledge between tasks with similar dynamics but different visual inputs.

Contribution

It introduces a transfer approach based on freezing learned internal dynamics and value functions within a low-dimensional embedding space, highlighting the effects of local minima and base model choice.

Findings

Frozen models can cause local minima affecting transfer quality.

Successful transfer depends heavily on the choice of the base model.

Converged embeddings enable faster transfer compared to learning from scratch.

Abstract

In this work we investigate a specific transfer learning approach for deep reinforcement learning in the context where the internal dynamics between two tasks are the same but the visual representations differ. We learn a low-dimensional encoding of the environment, meant to capture summarizing abstractions, from which the internal dynamics and value functions are learned. Transfer is then obtained by freezing the learned internal dynamics and value functions, thus reusing the shared low-dimensional embedding space. When retraining the encoder for transfer, we make several observations: (i) in some cases, there are local minima that have small losses but a mismatching embedding space, resulting in poor task performance and (ii) in the absence of local minima, the output of the encoder converges in our experiments to the same embedding space, which leads to a fast and efficient transfer as compared to learning from scratch. The local minima are caused by the reduced degree of freedom of the optimization process caused by the frozen models. We also find that the transfer performance is heavily reliant on the base model; some base models often result in a successful transfer, whereas other base models often result in a failing transfer.