Mesh-based Tools to Analyze Deep Reinforcement Learning Policies for Underactuated Biped Locomotion

TL;DR

This paper introduces a mesh-based analytical method to evaluate the stability and robustness of deep reinforcement learning policies for biped locomotion, offering a computationally efficient alternative to Monte Carlo simulations.

Contribution

The work develops a novel mesh-based approach to analyze policy robustness and stability, reducing computational costs compared to traditional simulation methods.

Findings

Mesh-based tools effectively assess robustness of policies.

Deep RL policies tend to contract to lower-dimensional manifolds.

Method provides a quantitative measure of policy stability.

Abstract

In this paper, we present a mesh-based approach to analyze stability and robustness of the policies obtained via deep reinforcement learning for various biped gaits of a five-link planar model. Intuitively, one would expect that including perturbations and/or other types of noise during training would likely result in more robustness of the resulting control policy. However, one would also like to have a quantitative and computationally-efficient means of evaluating the degree to which this might be so. Rather than relying on Monte Carlo simulations, which can become quite computationally burdensome in quantifying performance metrics, our goal is to provide more sophisticated tools to assess robustness properties of such policies. Our work is motivated by the twin hypotheses that contraction of dynamics, when achievable, can simplify the required complexity of a control policy and that control policies obtained via deep learning may therefore exhibit tendency to contract to lower-dimensional manifolds within the full state space, as a result. The tractability of our mesh-based tools in this work provides some evidence that this may be so.