Interpretable Reinforcement Learning for Robotics and Continuous Control

TL;DR

This paper introduces Interpretable Continuous Control Trees (ICCTs), a gradient-optimized, tree-based model for reinforcement learning that achieves high performance and interpretability in robotics and autonomous driving tasks.

Contribution

The paper presents ICCTs, a novel gradient-based, interpretable policy model that outperforms baselines and can be verified efficiently, advancing safe deployment in critical domains.

Findings

ICCTs outperform baselines by up to 33% in autonomous driving.

ICCTs reduce parameters by 300-600x compared to deep learning models.

ICCTs are rated easier to interpret and validate by end-users.

Abstract

Interpretability in machine learning is critical for the safe deployment of learned policies across legally-regulated and safety-critical domains. While gradient-based approaches in reinforcement learning have achieved tremendous success in learning policies for continuous control problems such as robotics and autonomous driving, the lack of interpretability is a fundamental barrier to adoption. We propose Interpretable Continuous Control Trees (ICCTs), a tree-based model that can be optimized via modern, gradient-based, reinforcement learning approaches to produce high-performing, interpretable policies. The key to our approach is a procedure for allowing direct optimization in a sparse decision-tree-like representation. We validate ICCTs against baselines across six domains, showing that ICCTs are capable of learning policies that parity or outperform baselines by up to 33% in autonomous driving scenarios while achieving a 300x-600x reduction in the number of parameters against deep learning baselines. We prove that ICCTs can serve as universal function approximators and display analytically that ICCTs can be verified in linear time. Furthermore, we deploy ICCTs in two realistic driving domains, based on interstate Highway-94 and 280 in the US. Finally, we verify ICCT's utility with end-users and find that ICCTs are rated easier to simulate, quicker to validate, and more interpretable than neural networks.