Mirror Descent Search and its Acceleration

TL;DR

This paper introduces Mirror Descent Search (MDS) and its accelerated variants, demonstrating their effectiveness in black-box optimization and reinforcement learning through faster convergence in experiments.

Contribution

The paper proposes novel MDS and AMDS algorithms, extending mirror descent techniques to reinforcement learning and black-box optimization with improved convergence.

Findings

MDS and AMDS converge faster than some state-of-the-art methods.

The methods are applicable to both black-box optimization and reinforcement learning.

The research clarifies the relationship between existing RL algorithms and mirror descent approaches.

Abstract

In recent years, attention has been focused on the relationship between black-box optimiza- tion problem and reinforcement learning problem. In this research, we propose the Mirror Descent Search (MDS) algorithm which is applicable both for black box optimization prob- lems and reinforcement learning problems. Our method is based on the mirror descent method, which is a general optimization algorithm. The contribution of this research is roughly twofold. We propose two essential algorithms, called MDS and Accelerated Mirror Descent Search (AMDS), and two more approximate algorithms: Gaussian Mirror Descent Search (G-MDS) and Gaussian Accelerated Mirror Descent Search (G-AMDS). This re- search shows that the advanced methods developed in the context of the mirror descent research can be applied to reinforcement learning problem. We also clarify the relationship between an existing reinforcement learning algorithm and our method. With two evaluation experiments, we show our proposed algorithms converge faster than some state-of-the-art methods.