Hypernetworks for Zero-shot Transfer in Reinforcement Learning

TL;DR

This paper introduces a hypernetwork-based approach for zero-shot transfer in reinforcement learning, enabling agents to generalize to unseen tasks by generating near-optimal policies from task parameters.

Contribution

The work presents a novel training objective and demonstrates how hypernetworks can generate policies for unseen tasks, advancing zero-shot transfer in RL.

Findings

Significant improvement over baseline methods in zero-shot transfer tasks

Effective generalization to new reward and transition dynamics

Applicable to continuous control tasks from DeepMind Control Suite

Abstract

In this paper, hypernetworks are trained to generate behaviors across a range of unseen task conditions, via a novel TD-based training objective and data from a set of near-optimal RL solutions for training tasks. This work relates to meta RL, contextual RL, and transfer learning, with a particular focus on zero-shot performance at test time, enabled by knowledge of the task parameters (also known as context). Our technical approach is based upon viewing each RL algorithm as a mapping from the MDP specifics to the near-optimal value function and policy and seek to approximate it with a hypernetwork that can generate near-optimal value functions and policies, given the parameters of the MDP. We show that, under certain conditions, this mapping can be considered as a supervised learning problem. We empirically evaluate the effectiveness of our method for zero-shot transfer to new reward and transition dynamics on a series of continuous control tasks from DeepMind Control Suite. Our method demonstrates significant improvements over baselines from multitask and meta RL approaches.