PaLI-3 Vision Language Models: Smaller, Faster, Stronger

TL;DR

PaLI-3 is a compact, efficient vision language model that outperforms larger models in multimodal tasks, especially in localization and text understanding, by leveraging contrastive pretraining with SigLIP.

Contribution

Introducing PaLI-3, a smaller and faster VLM that uses contrastive SigLIP pretraining, achieving state-of-the-art results with only 5 billion parameters.

Findings

SigLIP-based PaLI outperforms classification-pretrained models in multimodal benchmarks.

Scaling SigLIP to 2 billion parameters improves multilingual cross-modal retrieval.

PaLI-3 demonstrates superior performance despite being significantly smaller.

Abstract

This paper presents PaLI-3, a smaller, faster, and stronger vision language model (VLM) that compares favorably to similar models that are 10x larger. As part of arriving at this strong performance, we compare Vision Transformer (ViT) models pretrained using classification objectives to contrastively (SigLIP) pretrained ones. We find that, while slightly underperforming on standard image classification benchmarks, SigLIP-based PaLI shows superior performance across various multimodal benchmarks, especially on localization and visually-situated text understanding. We scale the SigLIP image encoder up to 2 billion parameters, and achieves a new state-of-the-art on multilingual cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles research on fundamental pieces of complex VLMs, and could fuel a new generation of scaled-up models.