SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks

TL;DR

SMOSE introduces a sparse, interpretable mixture-of-experts approach for reinforcement learning in continuous control, balancing performance with transparency by training specialized decisionmakers and distilling their logic into decision trees.

Contribution

The paper presents SMOSE, a novel sparse mixture-of-experts architecture with interpretable decisionmakers and routers, trained via reinforcement learning and distilled into decision trees for enhanced interpretability.

Findings

Outperforms recent interpretable baselines in MuJoCo environments.

Narrower performance gap with noninterpretable algorithms.

Effective training of sparse, interpretable controllers in complex tasks.

Abstract

Continuous control tasks often involve high-dimensional, dynamic, and non-linear environments. State-of-the-art performance in these tasks is achieved through complex closed-box policies that are effective, but suffer from an inherent opacity. Interpretable policies, while generally underperforming compared to their closed-box counterparts, advantageously facilitate transparent decision-making within automated systems. Hence, their usage is often essential for diagnosing and mitigating errors, supporting ethical and legal accountability, and fostering trust among stakeholders. In this paper, we propose SMOSE, a novel method to train sparsely activated interpretable controllers, based on a top-1 Mixture-of-Experts architecture. SMOSE combines a set of interpretable decisionmakers, trained to be experts in different basic skills, and an interpretable router that assigns tasks among the experts. The training is carried out via state-of-the-art Reinforcement Learning algorithms, exploiting load-balancing techniques to ensure fair expert usage. We then distill decision trees from the weights of the router, significantly improving the ease of interpretation. We evaluate SMOSE on six benchmark environments from MuJoCo: our method outperforms recent interpretable baselines and narrows the gap with noninterpretable state-of-the-art algorithms