Reward-Aware Proto-Representations in Reinforcement Learning

TL;DR

This paper introduces the default representation (DR), a reward-aware alternative to the successor representation (SR) in reinforcement learning, providing theoretical foundations and empirical benefits in exploration, transfer learning, and reward shaping.

Contribution

It develops the theoretical basis for the DR, extending it to function approximation, and demonstrates its advantages over SR in various RL tasks.

Findings

DR offers reward-aware behavior unlike SR.

DR outperforms SR in transfer learning and exploration.

Theoretical foundations for DR are established.

Abstract

In recent years, the successor representation (SR) has attracted increasing attention in reinforcement learning (RL), and it has been used to address some of its key challenges, such as exploration, credit assignment, and generalization. The SR can be seen as representing the underlying credit assignment structure of the environment by implicitly encoding its induced transition dynamics. However, the SR is reward-agnostic. In this paper, we discuss a similar representation that also takes into account the reward dynamics of the problem. We study the default representation (DR), a recently proposed representation with limited theoretical (and empirical) analysis. Here, we lay some of the theoretical foundation underlying the DR in the tabular case by (1) deriving dynamic programming and (2) temporal-difference methods to learn the DR, (3) characterizing the basis for the vector space of the DR, and (4) formally extending the DR to the function approximation case through default features. Empirically, we analyze the benefits of the DR in many of the settings in which the SR has been applied, including (1) reward shaping, (2) option discovery, (3) exploration, and (4) transfer learning. Our results show that, compared to the SR, the DR gives rise to qualitatively different, reward-aware behaviour and quantitatively better performance in several settings.