Time-Variant Variational Transfer for Value Functions

TL;DR

This paper introduces a variational transfer method for reinforcement learning that accounts for non-stationary, time-variant task distributions, with theoretical analysis and experimental validation across multiple environments.

Contribution

It proposes a novel transfer learning approach that leverages temporal structure in task distributions and provides finite-sample theoretical analysis.

Findings

The method effectively handles time-variant task distributions.

Theoretical comparison shows advantages over time-invariant approaches.

Experimental results demonstrate improved transfer performance across diverse environments.

Abstract

In most of the transfer learning approaches to reinforcement learning (RL) the distribution over the tasks is assumed to be stationary. Therefore, the target and source tasks are i.i.d. samples of the same distribution. In the context of this work, we consider the problem of transferring value functions through a variational method when the distribution that generates the tasks is time-variant, proposing a solution that leverages this temporal structure inherent in the task generating process. Furthermore, by means of a finite-sample analysis, the previously mentioned solution is theoretically compared to its time-invariant version. Finally, we will provide an experimental evaluation of the proposed technique with three distinct temporal dynamics in three different RL environments.