Token Activation Map to Visually Explain Multimodal LLMs

TL;DR

This paper introduces Token Activation Map (TAM), a novel method for explaining multimodal large language models by mitigating context interference and noise, enabling high-quality visualizations for model understanding and analysis.

Contribution

The paper proposes TAM, an innovative explanation technique for MLLMs that accounts for token interactions and reduces redundant activations, improving interpretability over existing methods.

Findings

TAM outperforms state-of-the-art explanation methods in visualization quality.

TAM effectively explains multiple tokens in MLLMs, unlike CAM.

The method is versatile for various visualization scenarios, including object localization and failure analysis.

Abstract

Multimodal large language models (MLLMs) are broadly empowering various fields. Despite their advancements, the explainability of MLLMs remains less explored, hindering deeper understanding, model credibility, and effective visualization. Unlike conventional vision models (e.g., CNNs, ViTs, CLIP) that produce a single output, MLLMs generate sequences of tokens progressively, where each generated token depends on the previous context. Therefore, earlier context tokens can introduce redundant activations that interfere with the explanation of later tokens beyond their original information. Existing studies often overlook this issue, but our observations reveal that these redundant correlations can significantly hurt the reliability of explanations. To address this, we propose an estimated causal inference method to mitigate the interference of context to achieve high-quality MLLM explanation, with a novel rank Gaussian filter to further reduce activation noises. We term this method Token Activation Map (TAM) to highlight the consideration of interactions between tokens. TAM also indicates that it excels at explaining multiple tokens of MLLM, which is different from the Class Activation Map (CAM) for a single prediction. Our TAM method significantly outperforms existing SoTA methods, showcasing high-quality visualization results that can be utilized for various scenarios, such as object localization, failure case analysis, video visualization, MLLMs visual comparison, and model understanding (e.g., color, shape, action, location, visual reasoning, multi-turn conversation, etc). The code is available atgithub.com/xmed-lab/TAM.