AutoQD: Automatic Discovery of Diverse Behaviors with Quality-Diversity Optimization

TL;DR

AutoQD introduces a theoretically grounded method to automatically generate behavioral descriptors for quality-diversity algorithms by embedding policy occupancy measures, enabling diverse policy discovery without predefined descriptors.

Contribution

The paper presents AutoQD, a novel approach that automatically generates behavioral descriptors using occupancy measure embeddings, eliminating the need for handcrafted descriptors in QD algorithms.

Findings

AutoQD effectively discovers diverse policies in continuous control tasks.

Embeddings converge to true MMD distances with increased samples and dimensions.

The method outperforms prior approaches relying on predefined behavioral descriptors.

Abstract

Quality-Diversity (QD) algorithms have shown remarkable success in discovering diverse, high-performing solutions, but rely heavily on hand-crafted behavioral descriptors that constrain exploration to predefined notions of diversity. Leveraging the equivalence between policies and occupancy measures, we present a theoretically grounded approach to automatically generate behavioral descriptors by embedding the occupancy measures of policies in Markov Decision Processes. Our method, AutoQD, leverages random Fourier features to approximate the Maximum Mean Discrepancy (MMD) between policy occupancy measures, creating embeddings whose distances reflect meaningful behavioral differences. A low-dimensional projection of these embeddings that captures the most behaviorally significant dimensions can then be used as behavioral descriptors for CMA-MAE, a state of the art blackbox QD method, to discover diverse policies. We prove that our embeddings converge to true MMD distances between occupancy measures as the number of sampled trajectories and embedding dimensions increase. Through experiments in multiple continuous control tasks we demonstrate AutoQD's ability in discovering diverse policies without predefined behavioral descriptors, presenting a well-motivated alternative to prior methods in unsupervised Reinforcement Learning and QD optimization. Our approach opens new possibilities for open-ended learning and automated behavior discovery in sequential decision making settings without requiring domain-specific knowledge. Source code is available at https://github.com/conflictednerd/autoqd-code.