The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss

TL;DR

This paper identifies a critical norm disparity issue in Pre-Norm multimodal large language models that hampers cross-modal fusion, and proposes a simple LayerNorm fix to improve overall performance.

Contribution

The work provides a formal analysis of the norm imbalance problem in Pre-Norm MLLMs and introduces a straightforward LayerNorm intervention to enhance multimodal and text-only tasks.

Findings

Norm disparity causes asymmetric update dynamics in MLLMs.

LayerNorm after visual projector improves multimodal benchmark performance.

Fixing norm imbalance also benefits text-only evaluation results.

Abstract

Multimodal Large Language Models (MLLMs), which couple pre-trained vision encoders and language models, have shown remarkable capabilities. However, their reliance on the ubiquitous Pre-Norm architecture introduces a subtle yet critical flaw: a severe norm disparity between the high-norm visual tokens and the low-norm text tokens. In this work, we present a formal theoretical analysis demonstrating that this imbalance is not a static issue. Instead, it induces an ``asymmetric update dynamic,'' where high-norm visual tokens exhibit a ``representational inertia,'' causing them to transform semantically much slower than their textual counterparts. This fundamentally impairs effective cross-modal feature fusion. Our empirical validation across a range of mainstream MLLMs confirms that this theoretical dynamic -- the persistence of norm disparity and the resulting asymmetric update rates -- is a prevalent phenomenon. Based on this insight, we propose a remarkably simple yet effective solution: inserting a single, carefully initialized LayerNorm layer after the visual projector to enforce norm alignment. Experiments conducted on the LLaVA-1.5 architecture show that this intervention yields significant performance gains not only on a wide suite of multimodal benchmarks but also, notably, on text-only evaluations such as MMLU, suggesting that resolving the architectural imbalance leads to a more holistically capable model.