LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning

TL;DR

LaViT introduces a novel framework that aligns latent visual thoughts to improve multi-modal reasoning by reconstructing visual semantics and attention trajectories, leading to significant performance gains.

Contribution

It proposes aligning latent visual thoughts instead of static embeddings, addressing the perception gap in multimodal models, and employs curriculum sensory gating for better grounding.

Findings

Achieves up to +16.9% gains on reasoning tasks.

Enables a 3B model to outperform larger models like GPT-4o.

Significantly improves visual grounding in multimodal reasoning.

Abstract

Current multimodal latent reasoning often relies on external supervision (e.g., auxiliary images), ignoring intrinsic visual attention dynamics. In this work, we identify a critical Perception Gap in distillation: student models frequently mimic a teacher's textual output while attending to fundamentally divergent visual regions, effectively relying on language priors rather than grounded perception. To bridge this, we propose LaViT, a framework that aligns latent visual thoughts rather than static embeddings. LaViT compels the student to autoregressively reconstruct the teacher's visual semantics and attention trajectories prior to text generation, employing a curriculum sensory gating mechanism to prevent shortcut learning. Extensive experiments show that LaViT significantly enhances visual grounding, achieving up to +16.9% gains on complex reasoning tasks and enabling a compact 3B model to outperform larger open-source variants and proprietary models like GPT-4o.