Objectives Are All You Need: Solving Deceptive Problems Without Explicit Diversity Maintenance

TL;DR

This paper introduces a method that solves deceptive optimization problems by optimizing multiple objectives derived from the environment, using lexicase selection, which implicitly maintains diversity without explicit diversity algorithms.

Contribution

The paper demonstrates that decomposing problems into many objectives and optimizing them with lexicase selection outperforms traditional diversity-maintenance algorithms like MAP-Elites in deceptive domains.

Findings

Objective decomposition improves performance on deceptive domains.

Lexicase selection implicitly maintains diversity without explicit mechanisms.

The approach is robust to different subaggregation techniques.

Abstract

Navigating deceptive domains has often been a challenge in machine learning due to search algorithms getting stuck at sub-optimal local optima. Many algorithms have been proposed to navigate these domains by explicitly maintaining diversity or equivalently promoting exploration, such as Novelty Search or other so-called Quality Diversity algorithms. In this paper, we present an approach with promise to solve deceptive domains without explicit diversity maintenance by optimizing a potentially large set of defined objectives. These objectives can be extracted directly from the environment by sub-aggregating the raw performance of individuals in a variety of ways. We use lexicase selection to optimize for these objectives as it has been shown to implicitly maintain population diversity. We compare this technique with a varying number of objectives to a commonly used quality diversity algorithm, MAP-Elites, on a set of discrete optimization as well as reinforcement learning domains with varying degrees of deception. We find that decomposing objectives into many objectives and optimizing them outperforms MAP-Elites on the deceptive domains that we explore. Furthermore, we find that this technique results in competitive performance on the diversity-focused metrics of QD-Score and Coverage, without explicitly optimizing for these things. Our ablation study shows that this technique is robust to different subaggregation techniques. However, when it comes to non-deceptive, or ``illumination" domains, quality diversity techniques generally outperform our objective-based framework with respect to exploration (but not exploitation), hinting at potential directions for future work.