Causal versus Marginal Shapley Values for Robotic Lever Manipulation Controlled using Deep Reinforcement Learning

TL;DR

This paper compares KernelSHAP and causal SHAP explanations for a robotic lever control task, demonstrating that incorporating domain knowledge and indirect effects yields explanations more aligned with human intuition.

Contribution

It introduces the use of causal SHAP with domain knowledge for robotic manipulation, highlighting the importance of accounting for indirect effects in explanations.

Findings

Causal SHAP better captures indirect feature effects.

Incorporating domain knowledge improves explanation quality.

Explanations align more with human intuition when using causal SHAP.

Abstract

We investigate the effect of including domain knowledge about a robotic system's causal relations when generating explanations. To this end, we compare two methods from explainable artificial intelligence, the popular KernelSHAP and the recent causal SHAP, on a deep neural network trained using deep reinforcement learning on the task of controlling a lever using a robotic manipulator. A primary disadvantage of KernelSHAP is that its explanations represent only the features' direct effects on a model's output, not considering the indirect effects a feature can have on the output by affecting other features. Causal SHAP uses a partial causal ordering to alter KernelSHAP's sampling procedure to incorporate these indirect effects. This partial causal ordering defines the causal relations between the features, and we specify this using domain knowledge about the lever control task. We show that enabling an explanation method to account for indirect effects and incorporating some domain knowledge can lead to explanations that better agree with human intuition. This is especially favorable for a real-world robotics task, where there is considerable causality at play, and in addition, the required domain knowledge is often handily available.