Quality Diversity for Robot Learning: Limitations and Future Directions

TL;DR

This paper critiques current Quality Diversity methods in robot learning, proposing a goal-conditioned policy approach with classical planning for better generalization and open-ended search, inspired by cognitive maps in neuroscience.

Contribution

It introduces a novel approach using a single goal-conditioned policy with classical planning, reducing complexity and enhancing generalization in QD for robot learning.

Findings

Single policy with classical planner achieves O(1) space complexity.

Approach generalizes to task variants.

Links QD with cognitive maps in neuroscience.

Abstract

Quality Diversity (QD) has shown great success in discovering high-performing, diverse policies for robot skill learning. While current benchmarks have led to the development of powerful QD methods, we argue that new paradigms must be developed to facilitate open-ended search and generalizability. In particular, many methods focus on learning diverse agents that each move to a different xy position in MAP-Elites-style bounded archives. Here, we show that such tasks can be accomplished with a single, goal-conditioned policy paired with a classical planner, achieving O(1) space complexity w.r.t. the number of policies and generalization to task variants. We hypothesize that this approach is successful because it extracts task-invariant structural knowledge by modeling a relational graph between adjacent cells in the archive. We motivate this view with emerging evidence from computational neuroscience and explore connections between QD and models of cognitive maps in human and other animal brains. We conclude with a discussion exploring the relationships between QD and cognitive maps, and propose future research directions inspired by cognitive maps towards future generalizable algorithms capable of truly open-ended search.