Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model

TL;DR

This paper introduces llama-nemoretriever-colembed, a unified text-image retrieval model that achieves state-of-the-art results on multiple benchmarks by combining a vision-language model with a ColBERT-style interaction and a two-stage training process.

Contribution

The paper presents a novel unified multimodal retrieval model with a modified architecture and a two-stage training strategy, achieving top performance on key benchmarks.

Findings

3B model scores NDCG@5 of 91.0 on ViDoRe V1

3B model scores NDCG@5 of 63.5 on ViDoRe V2

Analysis of trade-offs in storage and efficiency due to fine-grained retrieval mechanism

Abstract

Motivated by the growing demand for retrieval systems that operate across modalities, we introduce llama-nemoretriever-colembed, a unified text-image retrieval model that delivers state-of-the-art performance across multiple benchmarks. We release two model variants, 1B and 3B. The 3B model achieves state of the art performance, scoring NDCG@5 91.0 on ViDoRe V1 and 63.5 on ViDoRe V2, placing first on both leaderboards as of June 27, 2025. Our approach leverages the NVIDIA Eagle2 Vision-Language model (VLM), modifies its architecture by replacing causal attention with bidirectional attention, and integrates a ColBERT-style late interaction mechanism to enable fine-grained multimodal retrieval in a shared embedding space. While this mechanism delivers superior retrieval accuracy, it introduces trade-offs in storage and efficiency. We provide a comprehensive analysis of these trade-offs. Additionally, we adopt a two-stage training strategy to enhance the model's retrieval capabilities.