Configuration Path Control

TL;DR

This paper introduces a post-training stabilization method for reinforcement learning control policies by manipulating configuration paths, significantly improving their robustness against disturbances in a bipedal walker task.

Contribution

It presents a novel post-training stabilization technique based on configuration path control, enhancing RL policy robustness without retraining.

Findings

2-4 times increase in stability under perturbations

Effective stabilization applied post-training

Applicable to control policies in robotics

Abstract

Reinforcement learning methods often produce brittle policies -- policies that perform well during training, but generalize poorly beyond their direct training experience, thus becoming unstable under small disturbances. To address this issue, we propose a method for stabilizing a control policy in the space of configuration paths. It is applied post-training and relies purely on the data produced during training, as well as on an instantaneous control-matrix estimation. The approach is evaluated empirically on a planar bipedal walker subjected to a variety of perturbations. The control policies obtained via reinforcement learning are compared against their stabilized counterparts. Across different experiments, we find two- to four-fold increase in stability, when measured in terms of the perturbation amplitudes. We also provide a zero-dynamics interpretation of our approach.