Distributional Value Estimation Without Target Networks for Robust Quality-Diversity

TL;DR

This paper introduces QDHUAC, a target-free, distributional QD-RL algorithm that achieves high sample efficiency and stable training at high update-to-data ratios without target networks, enabling effective skill discovery in complex environments.

Contribution

The paper presents a novel target-free distributional critic approach for Quality-Diversity RL that improves sample efficiency and training stability at high update ratios.

Findings

Achieves competitive coverage and fitness with fewer environment steps.

Enables stable high-UTD training without target networks.

Demonstrates effectiveness on high-dimensional Brax environments.

Abstract

Quality-Diversity (QD) algorithms excel at discovering diverse repertoires of skills, but are hindered by poor sample efficiency and often require tens of millions of environment steps to solve complex locomotion tasks. Recent advances in Reinforcement Learning (RL) have shown that high Update-to-Data (UTD) ratios accelerate Actor-Critic learning. While effective, standard high-UTD algorithms typically utilise target networks to stabilise training. This requirement introduces a significant computational bottleneck, rendering them impractical for resource-intensive Quality-Diversity (QD) tasks where sample efficiency and rapid population adaptation are critical. In this paper, we introduce QDHUAC, a sample-efficient, target-free and distributional QD-RL algorithm that provides dense and low-variance gradient signals, which enables high-UTD training for Dominated Novelty Search whilst requiring an order of magnitude fewer environment steps. We demonstrate that our method enables stable training at high UTD ratios, achieving competitive coverage and fitness on high-dimensional Brax environments with an order of magnitude fewer samples than baselines. Our results suggest that combining target-free distributional critics with dominance-based selection is a key enabler for the next generation of sample-efficient evolutionary RL algorithms.