Towards Deep Robot Learning with Optimizer applicable to Non-stationary Problems

TL;DR

This paper introduces d-AmsGrad, a new optimizer designed for non-stationary data in robot learning, improving robustness and performance over existing optimizers like AmsGrad.

Contribution

The paper proposes d-AmsGrad, an improved optimizer that adapts to non-stationary problems by decaying the maximum second momentum, enhancing robot learning performance.

Findings

d-AmsGrad outperforms baseline optimizers in robotics tasks.

Decaying maximum second momentum improves learning in non-stationary environments.

The optimizer maintains the ability to reach global optima.

Abstract

This paper proposes a new optimizer for deep learning, named d-AmsGrad. In the real-world data, noise and outliers cannot be excluded from dataset to be used for learning robot skills. This problem is especially striking for robots that learn by collecting data in real time, which cannot be sorted manually. Several noise-robust optimizers have therefore been developed to resolve this problem, and one of them, named AmsGrad, which is a variant of Adam optimizer, has a proof of its convergence. However, in practice, it does not improve learning performance in robotics scenarios. This reason is hypothesized that most of robot learning problems are non-stationary, but AmsGrad assumes the maximum second momentum during learning to be stationarily given. In order to adapt to the non-stationary problems, an improved version, which slowly decays the maximum second momentum, is proposed. The proposed optimizer has the same capability of reaching the global optimum as baselines, and its performance outperformed that of the baselines in robotics problems.