Probabilistic Constrained Reinforcement Learning with Formal Interpretability

TL;DR

This paper introduces AWaVO, a novel probabilistic reinforcement learning method that enhances interpretability and guarantees convergence, demonstrated through simulations and quadrotor tasks, balancing performance and transparency.

Contribution

The paper proposes AWaVO, a new approach combining formal interpretability guarantees with probabilistic inference in reinforcement learning.

Findings

AWaVO achieves guaranteed interpretability with convergence guarantees.

It demonstrates high performance in simulation and quadrotor tasks.

Outperforms state-of-the-art benchmarks in interpretability and efficiency.

Abstract

Reinforcement learning can provide effective reasoning for sequential decision-making problems with variable dynamics. Such reasoning in practical implementation, however, poses a persistent challenge in interpreting the reward function and the corresponding optimal policy. Consequently, representing sequential decision-making problems as probabilistic inference can have considerable value, as, in principle, the inference offers diverse and powerful mathematical tools to infer the stochastic dynamics whilst suggesting a probabilistic interpretation of policy optimization. In this study, we propose a novel Adaptive Wasserstein Variational Optimization, namely AWaVO, to tackle these interpretability challenges. Our approach uses formal methods to achieve the interpretability for convergence guarantee, training transparency, and intrinsic decision-interpretation. To demonstrate its practicality, we showcase guaranteed interpretability with an optimal global convergence rate in simulation and in practical quadrotor tasks. In comparison with state-of-the-art benchmarks including TRPO-IPO, PCPO and CRPO, we empirically verify that AWaVO offers a reasonable trade-off between high performance and sufficient interpretability.