Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

TL;DR

This paper introduces a hierarchical reinforcement learning framework that enables agents to generalize zero-shot to new instructions and longer sequences by learning subtask correspondences and utilizing a meta controller with a novel update mechanism.

Contribution

It proposes a new RL problem setting for zero-shot task generalization, along with a hierarchical architecture and a novel neural meta controller for improved learning efficiency.

Findings

Effective generalization to unseen instructions

Successful handling of longer instruction sequences

Improved learning efficiency with the new neural architecture

Abstract

As a step towards developing zero-shot task generalization capabilities in reinforcement learning (RL), we introduce a new RL problem where the agent should learn to execute sequences of instructions after learning useful skills that solve subtasks. In this problem, we consider two types of generalizations: to previously unseen instructions and to longer sequences of instructions. For generalization over unseen instructions, we propose a new objective which encourages learning correspondences between similar subtasks by making analogies. For generalization over sequential instructions, we present a hierarchical architecture where a meta controller learns to use the acquired skills for executing the instructions. To deal with delayed reward, we propose a new neural architecture in the meta controller that learns when to update the subtask, which makes learning more efficient. Experimental results on a stochastic 3D domain show that the proposed ideas are crucial for generalization to longer instructions as well as unseen instructions.