Hypernetworks in Meta-Reinforcement Learning

TL;DR

This paper demonstrates that proper hypernetwork initialization significantly improves meta-reinforcement learning performance, introducing a novel initialization scheme that enhances generalization across tasks in robotics benchmarks.

Contribution

The authors identify hypernetwork initialization as crucial in meta-RL and propose a new, effective initialization method that outperforms existing approaches.

Findings

Hypernetwork initialization critically affects meta-RL performance.

The proposed initialization scheme matches or exceeds state-of-the-art results.

Hypernetworks improve task generalization in robotics benchmarks.

Abstract

Training a reinforcement learning (RL) agent on a real-world robotics task remains generally impractical due to sample inefficiency. Multi-task RL and meta-RL aim to improve sample efficiency by generalizing over a distribution of related tasks. However, doing so is difficult in practice: In multi-task RL, state of the art methods often fail to outperform a degenerate solution that simply learns each task separately. Hypernetworks are a promising path forward since they replicate the separate policies of the degenerate solution while also allowing for generalization across tasks, and are applicable to meta-RL. However, evidence from supervised learning suggests hypernetwork performance is highly sensitive to the initialization. In this paper, we 1) show that hypernetwork initialization is also a critical factor in meta-RL, and that naive initializations yield poor performance; 2) propose a novel hypernetwork initialization scheme that matches or exceeds the performance of a state-of-the-art approach proposed for supervised settings, as well as being simpler and more general; and 3) use this method to show that hypernetworks can improve performance in meta-RL by evaluating on multiple simulated robotics benchmarks.