RANKVIDEO: Reasoning Reranking for Text-to-Video Retrieval

TL;DR

RANKVIDEO introduces a reasoning-based reranker for text-to-video retrieval that explicitly reasons over query-video pairs, significantly improving retrieval accuracy on large-scale benchmarks through a novel training pipeline.

Contribution

It is the first to apply reasoning-based reranking to video retrieval, combining perception-grounded fine-tuning with a multi-objective reranking training pipeline.

Findings

Achieves 31% improvement on nDCG@10 over baseline methods.

Outperforms text-only and vision-language rerankers in accuracy.

Demonstrates efficiency and effectiveness on MultiVENT 2.0 benchmark.

Abstract

Reranking is a critical component of modern retrieval systems, which typically pair an efficient first-stage retriever with a more expressive model to refine results. While large reasoning models have driven rapid progress in text-centric reranking, reasoning-based reranking for video retrieval remains underexplored. To address this gap, we introduce RANKVIDEO, a reasoning-based reranker for video retrieval that explicitly reasons over query-video pairs using video content to assess relevance. RANKVIDEO is trained using a two-stage curriculum consisting of perception-grounded supervised fine-tuning followed by reranking training that combines pointwise, pairwise, and teacher confidence distillation objectives, and is supported by a data synthesis pipeline for constructing reasoning-intensive query-video pairs. Experiments on the large-scale MultiVENT 2.0 benchmark demonstrate that RANKVIDEO consistently improves retrieval performance within a two-stage framework, yielding an average improvement of 31% on nDCG@10 and outperforming text-only and vision-language reranking alternatives, while more efficient.