Experience Replay with Likelihood-free Importance Weights

TL;DR

This paper introduces a novel experience replay method that reweights past experiences based on their likelihood under the current policy's distribution, improving sample efficiency in deep reinforcement learning.

Contribution

It proposes a likelihood-free importance weighting scheme for experience replay, enhancing TD learning performance in deep RL algorithms.

Findings

Achieves superior sample complexity on OpenAI gym tasks

Improves performance of SAC and TD3 algorithms

Demonstrates effectiveness of likelihood-based experience prioritization

Abstract

The use of past experiences to accelerate temporal difference (TD) learning of value functions, or experience replay, is a key component in deep reinforcement learning. Prioritization or reweighting of important experiences has shown to improve performance of TD learning algorithms.In this work, we propose to reweight experiences based on their likelihood under the stationary distribution of the current policy. Using the corresponding reweighted TD objective, we implicitly encourage small approximation errors on the value function over frequently encountered states. We use a likelihood-free density ratio estimator over the replay buffer to assign the prioritization weights. We apply the proposed approach empirically on two competitive methods, Soft Actor Critic (SAC) and Twin Delayed Deep Deterministic policy gradient (TD3) -- over a suite of OpenAI gym tasks and achieve superior sample complexity compared to other baseline approaches.