Full-Gradient Successor Feature Representations

TL;DR

This paper introduces FG-SFRQL, a successor feature learning algorithm that minimizes the full Bellman error, leading to improved stability, sample efficiency, and transfer in reinforcement learning.

Contribution

It proposes a novel full-gradient approach for successor features, with theoretical convergence guarantees and empirical improvements over semi-gradient methods.

Findings

FG-SFRQL converges almost surely.

Full-gradient minimization improves transfer performance.

Method outperforms semi-gradient baselines in discrete and continuous tasks.

Abstract

Successor Features (SF) combined with Generalized Policy Improvement (GPI) provide a robust framework for transfer learning in Reinforcement Learning (RL) by decoupling environment dynamics from reward functions. However, standard SF learning methods typically rely on semi-gradient Temporal Difference (TD) updates. When combined with non-linear function approximation, semi-gradient methods lack robust convergence guarantees and can lead to instability, particularly in the multi-task setting where accurate feature estimation is critical for effective GPI. Inspired by Full Gradient DQN, we propose Full-Gradient Successor Feature Representations Q-Learning (FG-SFRQL), an algorithm that optimizes the successor features by minimizing the full Mean Squared Bellman Error. Unlike standard approaches, our method computes gradients with respect to parameters in both the online and target networks. We provide a theoretical proof of almost-sure convergence for FG-SFRQL and demonstrate empirically that minimizing the full residual leads to superior sample efficiency and transfer performance compared to semi-gradient baselines in both discrete and continuous domains.