Deep Successor Reinforcement Learning

TL;DR

This paper introduces Deep Successor Reinforcement Learning (DSR), a method that combines successor representations with deep learning to improve value function learning from raw observations, enabling better reward sensitivity and subgoal extraction.

Contribution

The paper presents DSR, a novel deep RL framework that integrates successor representations, enhancing reward sensitivity and subgoal identification from raw pixel inputs.

Findings

Effective in grid-world and Doom environments

Improves reward change sensitivity

Extracts subgoals from successor maps

Abstract

Learning robust value functions given raw observations and rewards is now possible with model-free and model-based deep reinforcement learning algorithms. There is a third alternative, called Successor Representations (SR), which decomposes the value function into two components -- a reward predictor and a successor map. The successor map represents the expected future state occupancy from any given state and the reward predictor maps states to scalar rewards. The value function of a state can be computed as the inner product between the successor map and the reward weights. In this paper, we present DSR, which generalizes SR within an end-to-end deep reinforcement learning framework. DSR has several appealing properties including: increased sensitivity to distal reward changes due to factorization of reward and world dynamics, and the ability to extract bottleneck states (subgoals) given successor maps trained under a random policy. We show the efficacy of our approach on two diverse environments given raw pixel observations -- simple grid-world domains (MazeBase) and the Doom game engine.