The Role of Exploration for Task Transfer in Reinforcement Learning

TL;DR

This paper examines how exploration strategies in reinforcement learning affect the efficiency of transferring knowledge between changing tasks, emphasizing the importance of anticipation in exploration for better transfer performance.

Contribution

It provides a review and taxonomy of exploration methods in RL, analyzing their roles in transfer learning and proposing directions for future research.

Findings

Exploration strategies significantly influence transfer efficiency.

Anticipatory exploration can improve adaptation to new tasks.

A taxonomy helps organize and compare exploration methods for transfer learning.

Abstract

The exploration--exploitation trade-off in reinforcement learning (RL) is a well-known and much-studied problem that balances greedy action selection with novel experience, and the study of exploration methods is usually only considered in the context of learning the optimal policy for a single learning task. However, in the context of online task transfer, where there is a change to the task during online operation, we hypothesize that exploration strategies that anticipate the need to adapt to future tasks can have a pronounced impact on the efficiency of transfer. As such, we re-examine the exploration--exploitation trade-off in the context of transfer learning. In this work, we review reinforcement learning exploration methods, define a taxonomy with which to organize them, analyze these methods' differences in the context of task transfer, and suggest avenues for future investigation.