Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning

TL;DR

This paper introduces a method using reward machine abstractions to improve transfer learning in deep reinforcement learning, enabling agents to better adapt to new tasks by leveraging symbolic representations of task structure.

Contribution

The paper proposes a novel approach to represent tasks with reward machines, facilitating transfer through symbolic abstractions and pre-planned transitions, which was not addressed in prior work.

Findings

Improved sample efficiency in transfer tasks

Enhanced few-shot learning capabilities

Better adaptation to unseen environments

Abstract

Recent studies show that deep reinforcement learning (DRL) agents tend to overfit to the task on which they were trained and fail to adapt to minor environment changes. To expedite learning when transferring to unseen tasks, we propose a novel approach to representing the current task using reward machines (RMs), state machine abstractions that induce subtasks based on the current task's rewards and dynamics. Our method provides agents with symbolic representations of optimal transitions from their current abstract state and rewards them for achieving these transitions. These representations are shared across tasks, allowing agents to exploit knowledge of previously encountered symbols and transitions, thus enhancing transfer. Empirical results show that our representations improve sample efficiency and few-shot transfer in a variety of domains.