Improving CMA-ES Convergence Speed, Efficiency, and Reliability in Noisy Robot Optimization Problems

TL;DR

This paper introduces AS-CMA, an adaptive sampling extension to CMA-ES, which improves convergence speed, efficiency, and reliability in noisy robot optimization tasks by dynamically allocating evaluation time based on difficulty predictions.

Contribution

The paper presents AS-CMA, a novel adaptive sampling method for CMA-ES that enhances optimization performance in noisy environments without extensive tuning.

Findings

AS-CMA converged in 98% of runs without tuning.

AS-CMA was 24-65% faster than static CMA-ES.

AS-CMA outperformed Bayesian optimization in complex landscapes.

Abstract

Experimental robot optimization often requires evaluating each candidate policy for seconds to minutes. The chosen evaluation time influences optimization because of a speed-accuracy tradeoff: shorter evaluations enable faster iteration, but are also more subject to noise. Here, we introduce a supplement to the CMA-ES optimization algorithm, named Adaptive Sampling CMA-ES (AS-CMA), which assigns sampling time to candidates based on predicted sorting difficulty, aiming to achieve consistent precision. We compared AS-CMA to CMA-ES and Bayesian optimization using a range of static sampling times in four simulated cost landscapes. AS-CMA converged on 98% of all runs without adjustment to its tunable parameter, and converged 24-65% faster and with 29-76% lower total cost than each landscape's best CMA-ES static sampling time. As compared to Bayesian optimization, AS-CMA converged more efficiently and reliably in complex landscapes, while in simpler landscapes, AS-CMA was less efficient but equally reliable. We deployed AS-CMA in an exoskeleton optimization experiment and found the optimizer's behavior was consistent with expectations. These results indicate that AS-CMA can improve optimization efficiency in the presence of noise while minimally affecting optimization setup complexity and tuning requirements.