UniIR: Training and Benchmarking Universal Multimodal Information Retrievers

TL;DR

UniIR is a versatile multimodal retrieval system trained on diverse datasets that can interpret instructions to perform various retrieval tasks across different media types, demonstrating strong generalization and establishing a new benchmark.

Contribution

The paper introduces UniIR, a unified instruction-guided multimodal retriever capable of handling eight retrieval tasks, and presents M-BEIR, a comprehensive benchmark for evaluation.

Findings

UniIR achieves robust performance across multiple datasets.

Multi-task training enhances generalization to new tasks.

M-BEIR provides a standardized evaluation platform.

Abstract

Existing information retrieval (IR) models often assume a homogeneous format, limiting their applicability to diverse user needs, such as searching for images with text descriptions, searching for a news article with a headline image, or finding a similar photo with a query image. To approach such different information-seeking demands, we introduce UniIR, a unified instruction-guided multimodal retriever capable of handling eight distinct retrieval tasks across modalities. UniIR, a single retrieval system jointly trained on ten diverse multimodal-IR datasets, interprets user instructions to execute various retrieval tasks, demonstrating robust performance across existing datasets and zero-shot generalization to new tasks. Our experiments highlight that multi-task training and instruction tuning are keys to UniIR's generalization ability. Additionally, we construct the M-BEIR, a multimodal retrieval benchmark with comprehensive results, to standardize the evaluation of universal multimodal information retrieval.