Explaining Multi-modal Large Language Models by Analyzing their Vision Perception

TL;DR

This paper introduces a new architecture for multi-modal large language models that enhances interpretability by linking vision embeddings to explain outputs, identify hallucinations, and analyze biases using saliency maps and adversarial perturbations.

Contribution

It proposes combining an open-world localization model with a MLLM to improve interpretability and enable detailed explanations of model outputs.

Findings

Enhanced interpretability with saliency maps

Ability to identify hallucinations and biases

Effective semantic adversarial perturbations

Abstract

Multi-modal Large Language Models (MLLMs) have demonstrated remarkable capabilities in understanding and generating content across various modalities, such as images and text. However, their interpretability remains a challenge, hindering their adoption in critical applications. This research proposes a novel approach to enhance the interpretability of MLLMs by focusing on the image embedding component. We combine an open-world localization model with a MLLM, thus creating a new architecture able to simultaneously produce text and object localization outputs from the same vision embedding. The proposed architecture greatly promotes interpretability, enabling us to design a novel saliency map to explain any output token, to identify model hallucinations, and to assess model biases through semantic adversarial perturbations.