Non-Parametric Stochastic Policy Gradient with Strategic Retreat for Non-Stationary Environment

TL;DR

This paper introduces a non-parametric, kernel-based policy learning method that adapts to non-stationary environments in robotics, outperforming traditional parametric methods like DDPG and TD3.

Contribution

It proposes a novel non-parametric kernel-based approach with adaptive windowing for dynamic environment adaptation, enhancing control policy learning in robotics.

Findings

Outperforms DDPG and TD3 in dynamic environments

Effective in high-dimensional RKHS spaces

Validated on multiple benchmarks with superior results

Abstract

In modern robotics, effectively computing optimal control policies under dynamically varying environments poses substantial challenges to the off-the-shelf parametric policy gradient methods, such as the Deep Deterministic Policy Gradient (DDPG) and Twin Delayed Deep Deterministic policy gradient (TD3). In this paper, we propose a systematic methodology to dynamically learn a sequence of optimal control policies non-parametrically, while autonomously adapting with the constantly changing environment dynamics. Specifically, our non-parametric kernel-based methodology embeds a policy distribution as the features in a non-decreasing Euclidean space, therefore allowing its search space to be defined as a very high (possible infinite) dimensional RKHS (Reproducing Kernel Hilbert Space). Moreover, by leveraging the similarity metric computed in RKHS, we augmented our non-parametric learning with the technique of AdaptiveH- adaptively selecting a time-frame window of finishing the optimal part of whole action-sequence sampled on some preceding observed state. To validate our proposed approach, we conducted extensive experiments with multiple classic benchmarks and one simulated robotics benchmark equipped with dynamically changing environments. Overall, our methodology has outperformed the well-established DDPG and TD3 methodology by a sizeable margin in terms of learning performance.