SHARCS: Shared Concept Space for Explainable Multimodal Learning

TL;DR

SHARCS introduces a shared concept space for explainable multimodal learning, enabling interpretable cross-modal analysis, improved predictive performance, and effective handling of missing modalities across diverse data types.

Contribution

It proposes a novel unified concept manifold that maps heterogeneous modalities into an interpretable space, enhancing explainability and performance in multimodal tasks.

Findings

SHARCS improves interpretability of multimodal models.

It outperforms existing methods in missing modality retrieval.

The approach is model-agnostic and versatile across modalities.

Abstract

Multimodal learning is an essential paradigm for addressing complex real-world problems, where individual data modalities are typically insufficient to accurately solve a given modelling task. While various deep learning approaches have successfully addressed these challenges, their reasoning process is often opaque; limiting the capabilities for a principled explainable cross-modal analysis and any domain-expert intervention. In this paper, we introduce SHARCS (SHARed Concept Space) -- a novel concept-based approach for explainable multimodal learning. SHARCS learns and maps interpretable concepts from different heterogeneous modalities into a single unified concept-manifold, which leads to an intuitive projection of semantically similar cross-modal concepts. We demonstrate that such an approach can lead to inherently explainable task predictions while also improving downstream predictive performance. Moreover, we show that SHARCS can operate and significantly outperform other approaches in practically significant scenarios, such as retrieval of missing modalities and cross-modal explanations. Our approach is model-agnostic and easily applicable to different types (and number) of modalities, thus advancing the development of effective, interpretable, and trustworthy multimodal approaches.