Multimodal Coherent Explanation Generation of Robot Failures

TL;DR

This paper presents a method for generating coherent multimodal explanations of robot failures by assessing and refining the logical consistency across different explanation modalities, improving interpretability for non-expert users.

Contribution

It introduces a coherence assessment approach using a neural network fine-tuned for textual entailment to ensure multimodal explanations are logically consistent.

Findings

Neural network fine-tuned for textual entailment effectively assesses explanation coherence.

The proposed method improves the clarity and consistency of multimodal robot failure explanations.

Coherent explanations help non-expert users better understand robot capabilities and limitations.

Abstract

The explainability of a robot's actions is crucial to its acceptance in social spaces. Explaining why a robot fails to complete a given task is particularly important for non-expert users to be aware of the robot's capabilities and limitations. So far, research on explaining robot failures has only considered generating textual explanations, even though several studies have shown the benefits of multimodal ones. However, a simple combination of multiple modalities may lead to semantic incoherence between the information across different modalities - a problem that is not well-studied. An incoherent multimodal explanation can be difficult to understand, and it may even become inconsistent with what the robot and the human observe and how they perform reasoning with the observations. Such inconsistencies may lead to wrong conclusions about the robot's capabilities. In this paper, we introduce an approach to generate coherent multimodal explanations by checking the logical coherence of explanations from different modalities, followed by refinements as required. We propose a classification approach for coherence assessment, where we evaluate if an explanation logically follows another. Our experiments suggest that fine-tuning a neural network that was pre-trained to recognize textual entailment, performs well for coherence assessment of multimodal explanations. Code & data: https://pradippramanick.github.io/coherent-explain/.