A Concept-Based Explainability Framework for Large Multimodal Models

TL;DR

This paper introduces a novel dictionary learning-based framework to interpret large multimodal models by extracting semantically grounded multimodal concepts, enhancing understanding of internal representations.

Contribution

It proposes a new interpretability method for LMMs using dictionary learning to identify and ground multimodal concepts within token representations.

Findings

Extracted semantically meaningful multimodal concepts

Improved interpretability of LMM internal representations

Demonstrated usefulness of concepts in understanding model behavior

Abstract

Large multimodal models (LMMs) combine unimodal encoders and large language models (LLMs) to perform multimodal tasks. Despite recent advancements towards the interpretability of these models, understanding internal representations of LMMs remains largely a mystery. In this paper, we present a novel framework for the interpretation of LMMs. We propose a dictionary learning based approach, applied to the representation of tokens. The elements of the learned dictionary correspond to our proposed concepts. We show that these concepts are well semantically grounded in both vision and text. Thus we refer to these as ``multi-modal concepts''. We qualitatively and quantitatively evaluate the results of the learnt concepts. We show that the extracted multimodal concepts are useful to interpret representations of test samples. Finally, we evaluate the disentanglement between different concepts and the quality of grounding concepts visually and textually. Our code is publicly available at https://github.com/mshukor/xl-vlms