MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model

TL;DR

This paper explores the internal neuron mechanisms of multimodal large language models, identifying domain-specific neurons and proposing a three-stage processing mechanism to enhance understanding and manipulation of cross-domain features.

Contribution

It introduces the concept of domain-specific neurons in MLLMs and proposes a three-stage mechanism for processing projected image features, verified through extensive experiments.

Findings

Current MLLMs have limited utilization of domain-specific information.

Manipulating domain-specific neurons can change accuracy by up to 10%.

The study provides insights for developing more comprehensive cross-domain MLLMs.

Abstract

Projecting visual features into word embedding space has become a significant fusion strategy adopted by Multimodal Large Language Models (MLLMs). However, its internal mechanisms have yet to be explored. Inspired by multilingual research, we identify domain-specific neurons in multimodal large language models. Specifically, we investigate the distribution of domain-specific neurons and the mechanism of how MLLMs process features from diverse domains. Furthermore, we propose a three-stage mechanism for language model modules in MLLMs when handling projected image features, and verify this hypothesis using logit lens. Extensive experiments indicate that while current MLLMs exhibit Visual Question Answering (VQA) capability, they may not fully utilize domain-specific information. Manipulating domain-specific neurons properly will result in a 10% change of accuracy at most, shedding light on the development of cross-domain, all-encompassing MLLMs in the future. The source code is available at https://github.com/Z1zs/MMNeuron.