GIFT: Global stabilisation via Intrinsic Fine Tuning

TL;DR

GIFT is a training framework that enhances the global stability of deep reinforcement learning policies, making them more suitable for real-world control applications by directly optimizing stability without sacrificing performance.

Contribution

GIFT introduces a novel method to improve the stability of deep RL policies through intrinsic fine tuning, addressing a key limitation for real-world deployment.

Findings

GIFT increases the stability of control policies.

GIFT maintains task performance while improving stability.

The framework is general-purpose and applicable to various policies.

Abstract

Deep reinforcement learning policies achieve strong performance in complex continuous control environments with nonlinear contact forces. However, these policies often produce chaotic state dynamics, with trivially small changes to the initial conditions significantly impacting the long-term behaviour of the control system. This high sensitivity to initial conditions limits the application of Deep RL to real-world control systems where performance and stability guarantees are often required. To address this issue, we propose Global stabilisation via Intrinsic Fine Tuning (GIFT), a general-purpose training framework which directly optimises the global stability of existing high-performing deep RL policies using a custom reward function. We demonstrate that GIFT increase the stability of the control interaction while maintaining comparable task performance, thereby improving the suitability of deep RL policies for real-world control systems.