Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models

TL;DR

JARVIS introduces a self-supervised visual learning framework for multimodal large language models, improving their visual reasoning and understanding by leveraging vision foundation models and structural regularities without relying solely on language supervision.

Contribution

The paper presents JARVIS, a novel self-supervised visual enhancement method that integrates I-JEPA into MLLMs, significantly boosting visual task performance without harming multimodal reasoning.

Findings

Improved performance on vision-centric benchmarks across various MLLMs.

Enhanced visual understanding without degrading multimodal reasoning.

Effective use of frozen vision models as context and target encoders.

Abstract

Multimodal Large Language Models (MLLMs) have recently demonstrated impressive capabilities in connecting vision and language, yet their proficiency in fundamental visual reasoning tasks remains limited. This limitation can be attributed to the fact that MLLMs learn visual understanding primarily from textual descriptions, which constitute a subjective and inherently incomplete supervisory signal. Furthermore, the modest scale of multimodal instruction tuning compared to massive text-only pre-training leads MLLMs to overfit language priors while overlooking visual details. To address these issues, we introduce JARVIS, a JEPA-inspired framework for self-supervised visual enhancement in MLLMs. Specifically, we integrate the I-JEPA learning paradigm into the standard vision-language alignment pipeline of MLLMs training. Our approach leverages frozen vision foundation models as context and target encoders, while training the predictor, implemented as the early layers of an LLM, to learn structural and semantic regularities from images without relying exclusively on language supervision. Extensive experiments on standard MLLM benchmarks show that JARVIS consistently improves performance on vision-centric benchmarks across different LLM families, without degrading multimodal reasoning abilities. Our source code is publicly available at: https://github.com/aimagelab/JARVIS.