FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions

TL;DR

FineViT is a new vision encoder that enhances fine-grained perception by training on dense recaptions and high-resolution data, significantly improving zero-shot recognition and retrieval in multimodal models.

Contribution

Introducing FineViT, a vision encoder trained with dense recaptions and a progressive paradigm to unlock detailed visual understanding in multimodal models.

Findings

Achieves state-of-the-art zero-shot recognition performance.

Outperforms existing multimodal visual encoders like SigLIP2 and Qwen-ViT.

Excels in long-context retrieval tasks.

Abstract

While Multimodal Large Language Models (MLLMs) have experienced rapid advancements, their visual encoders frequently remain a performance bottleneck. Conventional CLIP-based encoders struggle with dense spatial tasks due to the loss of visual details caused by low-resolution pretraining and the reliance on noisy, coarse web-crawled image-text pairs. To overcome these limitations, we introduce FineViT, a novel vision encoder specifically designed to unlock fine-grained perception. By replacing coarse web data with dense recaptions, we systematically mitigate information loss through a progressive training paradigm.: first, the encoder is trained from scratch at a high native resolution on billions of global recaptioned image-text pairs, establishing a robust, detail rich semantic foundation. Subsequently, we further enhance its local perception through LLM alignment, utilizing our curated FineCap-450M dataset that comprises over $450$ million high quality local captions. Extensive experiments validate the effectiveness of the progressive strategy. FineViT achieves state-of-the-art zero-shot recognition and retrieval performance, especially in long-context retrieval, and consistently outperforms multimodal visual encoders such as SigLIP2 and Qwen-ViT when integrated into MLLMs. We hope FineViT could serve as a powerful new baseline for fine-grained visual perception.