Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning

TL;DR

This paper explores how to effectively approximate gradients in differentiable quality diversity algorithms for reinforcement learning, enabling the training of diverse, high-performing agent policies in environments where exact gradients are unavailable.

Contribution

It introduces two gradient approximation variants for DQD algorithms in RL, demonstrating their effectiveness on locomotion tasks and highlighting current limitations.

Findings

One variant matches state-of-the-art results in QD and RL.

The other variant performs well on multiple locomotion tasks.

Results reveal limitations of current DQD algorithms with approximate gradients.

Abstract

Consider the problem of training robustly capable agents. One approach is to generate a diverse collection of agent polices. Training can then be viewed as a quality diversity (QD) optimization problem, where we search for a collection of performant policies that are diverse with respect to quantified behavior. Recent work shows that differentiable quality diversity (DQD) algorithms greatly accelerate QD optimization when exact gradients are available. However, agent policies typically assume that the environment is not differentiable. To apply DQD algorithms to training agent policies, we must approximate gradients for performance and behavior. We propose two variants of the current state-of-the-art DQD algorithm that compute gradients via approximation methods common in reinforcement learning (RL). We evaluate our approach on four simulated locomotion tasks. One variant achieves results comparable to the current state-of-the-art in combining QD and RL, while the other performs comparably in two locomotion tasks. These results provide insight into the limitations of current DQD algorithms in domains where gradients must be approximated. Source code is available at https://github.com/icaros-usc/dqd-rl