Latent Denoising Improves Visual Alignment in Large Multimodal Models

TL;DR

This paper introduces a latent denoising framework that enhances visual representation and reasoning in large multimodal models by training them to recover clean visual features from corrupted inputs, leading to improved robustness and understanding.

Contribution

The authors propose a novel latent denoising method that improves internal visual feature alignment and multimodal understanding in LMMs without increasing inference complexity.

Findings

Consistent improvement in visual understanding and reasoning benchmarks.

Enhanced robustness to common image corruptions.

Better compositional robustness on benchmarks like NaturalBench.

Abstract

Large Multimodal Models (LMMs) such as LLaVA are typically trained with an autoregressive language modeling objective, providing only indirect supervision to visual tokens. This often yields weak internal visual representations and brittle behavior under distribution shift. Inspired by recent progress on latent denoising for learning high-quality visual tokenizers, we show that the same principle provides an effective form of visual supervision for improving internal visual feature alignment and multimodal understanding in LMMs. We propose a latent denoising framework that corrupts projected visual tokens using a saliency-aware mixture of masking and Gaussian noising. The LMM is trained to denoise these corrupted tokens by recovering clean teacher patch features from hidden states at a selected intermediate LLM layer using a decoder. To prevent representation collapse, our framework also preserves the teacher's intra-image similarity structure and applies intra-image contrastive patch distillation. During inference, corruption and auxiliary heads are disabled, introducing no additional inference-time overhead. Across a broad suite of standard multimodal benchmarks, our method consistently improves visual understanding and reasoning over strong baselines, and yields clear gains on compositional robustness benchmarks (e.g., NaturalBench). Moreover, under ImageNet-C-style non-adversarial common corruptions applied to benchmark images, our method maintains higher accuracy and exhibits reduced degradation at both moderate and severe corruption levels. Our code is available at https://github.com/dhruvashp/latent-denoising-for-lmms.