Explicitly Encouraging Low Fractional Dimensional Trajectories Via Reinforcement Learning

TL;DR

This paper introduces a method to encourage reinforcement learning agents to follow lower-dimensional trajectories using fractal geometry, improving robustness and interpretability of control policies.

Contribution

It proposes a novel reward modification technique that influences the fractional dimensionality of trajectories in model-free reinforcement learning, addressing the curse of dimensionality.

Findings

Trajectories can be driven to lower fractional dimensions using reward shaping.

Lower-dimensional trajectories are more robust to noise and disturbances.

The method is verified across multiple systems with consistent results.

Abstract

A key limitation in using various modern methods of machine learning in developing feedback control policies is the lack of appropriate methodologies to analyze their long-term dynamics, in terms of making any sort of guarantees (even statistically) about robustness. The central reasons for this are largely due to the so-called curse of dimensionality, combined with the black-box nature of the resulting control policies themselves. This paper aims at the first of these issues. Although the full state space of a system may be quite large in dimensionality, it is a common feature of most model-based control methods that the resulting closed-loop systems demonstrate dominant dynamics that are rapidly driven to some lower-dimensional sub-space within. In this work we argue that the dimensionality of this subspace is captured by tools from fractal geometry, namely various notions of a fractional dimension. We then show that the dimensionality of trajectories induced by model free reinforcement learning agents can be influenced adding a post processing function to the agents reward signal. We verify that the dimensionality reduction is robust to noise being added to the system and show that that the modified agents are more actually more robust to noise and push disturbances in general for the systems we examined.