Approximating a deep reinforcement learning docking agent using linear model trees

TL;DR

This paper demonstrates that linear model trees can effectively approximate deep reinforcement learning policies for autonomous surface vehicle docking, providing transparent and real-time explanations of control decisions.

Contribution

The novel use of linear model trees to approximate DNN policies in autonomous docking, enabling explainability and real-time interpretability.

Findings

LMTs accurately approximate DNN control policies.

LMTs provide transparent feature attributions.

LMTs operate efficiently in real-time scenarios.

Abstract

Deep reinforcement learning has led to numerous notable results in robotics. However, deep neural networks (DNNs) are unintuitive, which makes it difficult to understand their predictions and strongly limits their potential for real-world applications due to economic, safety, and assurance reasons. To remedy this problem, a number of explainable AI methods have been presented, such as SHAP and LIME, but these can be either be too costly to be used in real-time robotic applications or provide only local explanations. In this paper, the main contribution is the use of a linear model tree (LMT) to approximate a DNN policy, originally trained via proximal policy optimization(PPO), for an autonomous surface vehicle with five control inputs performing a docking operation. The two main benefits of the proposed approach are: a) LMTs are transparent which makes it possible to associate directly the outputs (control actions, in our case) with specific values of the input features, b) LMTs are computationally efficient and can provide information in real-time. In our simulations, the opaque DNN policy controls the vehicle and the LMT runs in parallel to provide explanations in the form of feature attributions. Our results indicate that LMTs can be a useful component within digital assurance frameworks for autonomous ships.