Covariance Matrix Adaptation Evolution Strategy for Low Effective Dimensionality

TL;DR

This paper introduces CMA-ES-LED, an enhanced version of CMA-ES designed to effectively handle low effective dimensionality in high-dimensional black-box optimization problems by estimating and focusing on intrinsic dimensions.

Contribution

The paper proposes CMA-ES-LED, which incorporates dimension effectiveness estimation and refined step-size adaptations to improve optimization performance under LED conditions.

Findings

CMA-ES-LED outperforms standard CMA-ES on LED benchmark functions.

Effective dimension estimation improves search efficiency.

Refined step-size adaptation enhances convergence in low effective dimensionality scenarios.

Abstract

Despite the state-of-the-art performance of the covariance matrix adaptation evolution strategy (CMA-ES), high-dimensional black-box optimization problems are challenging tasks. Such problems often involve a property called low effective dimensionality (LED), in which the objective function is formulated with redundant dimensions relative to the intrinsic objective function and a rotation transformation of the search space. The CMA-ES suffers from LED for two reasons: the default hyperparameter setting is determined by the total number of dimensions, and the norm calculations in step-size adaptations are performed including elements on the redundant dimensions. In this paper, we incorporate countermeasures for LED into the CMA-ES and propose CMA-ES-LED. We tackle with the rotation transformation using the eigenvectors of the covariance matrix. We estimate the effectiveness of each dimension in the rotated search space using the element-wise signal-to-noise ratios of the mean vector update and the rank-$\mu$ update, both of which updates can be explained as the natural gradient ascent. Then, we adapt the hyperparameter using the estimated number of effective dimensions. In addition, we refine the cumulative step-size adaptation and the two-point step-size adaptation to measure the norms only on the effective dimensions. The experimental results show the CMA-ES-LED outperforms the CMA-ES on benchmark functions with LED.