Meta-Reinforcement Learning for Heuristic Planning

TL;DR

This paper introduces an information-theoretic task selection method for meta-reinforcement learning, which enhances training efficiency and test performance by choosing relevant and diverse training tasks.

Contribution

The paper proposes ITTS, a novel task selection algorithm that improves meta-RL training by optimizing task relevance and diversity, leading to better generalization.

Findings

ITTS improves meta-RL performance across multiple experiments.

Task relevance and diversity are crucial for effective meta-RL training.

ITTS outperforms random and other selection strategies.

Abstract

In Meta-Reinforcement Learning (meta-RL) an agent is trained on a set of tasks to prepare for and learn faster in new, unseen, but related tasks. The training tasks are usually hand-crafted to be representative of the expected distribution of test tasks and hence all used in training. We show that given a set of training tasks, learning can be both faster and more effective (leading to better performance in the test tasks), if the training tasks are appropriately selected. We propose a task selection algorithm, Information-Theoretic Task Selection (ITTS), based on information theory, which optimizes the set of tasks used for training in meta-RL, irrespectively of how they are generated. The algorithm establishes which training tasks are both sufficiently relevant for the test tasks, and different enough from one another. We reproduce different meta-RL experiments from the literature and show that ITTS improves the final performance in all of them.