Fisher-Guided Selective Forgetting: Mitigating The Primacy Bias in Deep Reinforcement Learning

TL;DR

This paper investigates the primacy bias in deep reinforcement learning using Fisher Information, and introduces Fisher-Guided Selective Forgetting to improve learning by mitigating early experience overfitting.

Contribution

It provides a FIM-based framework to understand primacy bias and proposes a novel method for selective forgetting to enhance DRL performance.

Findings

FGSF outperforms baselines in complex environments

Primacy bias affects actor and critic networks differently

Replay ratios and noise injection influence primacy bias effects

Abstract

Deep Reinforcement Learning (DRL) systems often tend to overfit to early experiences, a phenomenon known as the primacy bias (PB). This bias can severely hinder learning efficiency and final performance, particularly in complex environments. This paper presents a comprehensive investigation of PB through the lens of the Fisher Information Matrix (FIM). We develop a framework characterizing PB through distinct patterns in the FIM trace, identifying critical memorization and reorganization phases during learning. Building on this understanding, we propose Fisher-Guided Selective Forgetting (FGSF), a novel method that leverages the geometric structure of the parameter space to selectively modify network weights, preventing early experiences from dominating the learning process. Empirical results across DeepMind Control Suite (DMC) environments show that FGSF consistently outperforms baselines, particularly in complex tasks. We analyze the different impacts of PB on actor and critic networks, the role of replay ratios in exacerbating the effect, and the effectiveness of even simple noise injection methods. Our findings provide a deeper understanding of PB and practical mitigation strategies, offering a FIM-based geometric perspective for advancing DRL.