Empirical Evaluation of Contextual Policy Search with a Comparison-based Surrogate Model and Active Covariance Matrix Adaptation

TL;DR

This paper empirically evaluates enhancements to contextual policy search algorithms, specifically comparing a surrogate model-based approach and an active covariance update, demonstrating significant sample-efficiency improvements.

Contribution

It introduces and empirically assesses two extensions of C-CMA-ES: a comparison-based surrogate model and an active covariance matrix update.

Findings

Impressive sample-efficiency improvements with the proposed methods.

Extensions show significant benefits in optimization performance.

Relevance varies depending on robotic application context.

Abstract

Contextual policy search (CPS) is a class of multi-task reinforcement learning algorithms that is particularly useful for robotic applications. A recent state-of-the-art method is Contextual Covariance Matrix Adaptation Evolution Strategies (C-CMA-ES). It is based on the standard black-box optimization algorithm CMA-ES. There are two useful extensions of CMA-ES that we will transfer to C-CMA-ES and evaluate empirically: ACM-ES, which uses a comparison-based surrogate model, and aCMA-ES, which uses an active update of the covariance matrix. We will show that improvements with these methods can be impressive in terms of sample-efficiency, although this is not relevant any more for the robotic domain.