How Should an Agent Practice?

TL;DR

This paper introduces a meta-gradient method for learning intrinsic rewards that guide an agent's practice periods, improving performance in environments where practice differs from evaluation conditions.

Contribution

It proposes a novel meta-gradient approach to learn intrinsic practice rewards, enabling agents to better utilize practice environments for improved performance.

Findings

Learning in practice improves match performance.

Method effective in grid world, Pong, and PacMan.

Practice rewards adapt to environment differences.

Abstract

We present a method for learning intrinsic reward functions to drive the learning of an agent during periods of practice in which extrinsic task rewards are not available. During practice, the environment may differ from the one available for training and evaluation with extrinsic rewards. We refer to this setup of alternating periods of practice and objective evaluation as practice-match, drawing an analogy to regimes of skill acquisition common for humans in sports and games. The agent must effectively use periods in the practice environment so that performance improves during matches. In the proposed method the intrinsic practice reward is learned through a meta-gradient approach that adapts the practice reward parameters to reduce the extrinsic match reward loss computed from matches. We illustrate the method on a simple grid world, and evaluate it in two games in which the practice environment differs from match: Pong with practice against a wall without an opponent, and PacMan with practice in a maze without ghosts. The results show gains from learning in practice in addition to match periods over learning in matches only.