MEGL: Multimodal Explanation-Guided Learning

TL;DR

MEGL introduces a multimodal explanation-guided learning framework that combines visual and textual explanations to improve AI interpretability and classification accuracy, especially when visual annotations are incomplete.

Contribution

The paper proposes a novel MEGL framework that integrates visual and textual explanations, including a saliency-driven grounding method and a distribution consistency loss, advancing multimodal interpretability.

Findings

MEGL outperforms previous methods in accuracy and explanation quality.

The approach effectively learns from incomplete multimodal supervision.

Experimental validation on Object-ME and Action-ME datasets confirms its effectiveness.

Abstract

Explaining the decision-making processes of Artificial Intelligence (AI) models is crucial for addressing their "black box" nature, particularly in tasks like image classification. Traditional eXplainable AI (XAI) methods typically rely on unimodal explanations, either visual or textual, each with inherent limitations. Visual explanations highlight key regions but often lack rationale, while textual explanations provide context without spatial grounding. Further, both explanation types can be inconsistent or incomplete, limiting their reliability. To address these challenges, we propose a novel Multimodal Explanation-Guided Learning (MEGL) framework that leverages both visual and textual explanations to enhance model interpretability and improve classification performance. Our Saliency-Driven Textual Grounding (SDTG) approach integrates spatial information from visual explanations into textual rationales, providing spatially grounded and contextually rich explanations. Additionally, we introduce Textual Supervision on Visual Explanations to align visual explanations with textual rationales, even in cases where ground truth visual annotations are missing. A Visual Explanation Distribution Consistency loss further reinforces visual coherence by aligning the generated visual explanations with dataset-level patterns, enabling the model to effectively learn from incomplete multimodal supervision. We validate MEGL on two new datasets, Object-ME and Action-ME, for image classification with multimodal explanations. Experimental results demonstrate that MEGL outperforms previous approaches in prediction accuracy and explanation quality across both visual and textual domains. Our code will be made available upon the acceptance of the paper.