Reasoning Text-to-Video Retrieval via Digital Twin Video Representations and Large Language Models

TL;DR

This paper introduces a reasoning-based text-to-video retrieval method using digital twin scene representations and large language models, enabling implicit query understanding and object-level grounding, significantly improving retrieval accuracy.

Contribution

It proposes a novel digital twin video representation combined with a two-stage reasoning framework for implicit query retrieval, outperforming existing methods and establishing new benchmarks.

Findings

Achieves 81.2% R@1 on ReasonT2VBench-135, surpassing baselines by over 50%

Maintains high performance with 81.7% R@1 on extended datasets

Sets new state-of-the-art results on MSR-VTT, MSVD, and VATEX benchmarks.

Abstract

The goal of text-to-video retrieval is to search large databases for relevant videos based on text queries. Existing methods have progressed to handling explicit queries where the visual content of interest is described explicitly; however, they fail with implicit queries where identifying videos relevant to the query requires reasoning. We introduce reasoning text-to-video retrieval, a paradigm that extends traditional retrieval to process implicit queries through reasoning while providing object-level grounding masks that identify which entities satisfy the query conditions. Instead of relying on vision-language models directly, we propose representing video content as digital twins, i.e., structured scene representations that decompose salient objects through specialist vision models. This approach is beneficial because it enables large language models to reason directly over long-horizon video content without visual token compression. Specifically, our two-stage framework first performs compositional alignment between decomposed sub-queries and digital twin representations for candidate identification, then applies large language model-based reasoning with just-in-time refinement that invokes additional specialist models to address information gaps. We construct a benchmark of 447 manually created implicit queries with 135 videos (ReasonT2VBench-135) and another more challenging version of 1000 videos (ReasonT2VBench-1000). Our method achieves 81.2% R@1 on ReasonT2VBench-135, outperforming the strongest baseline by greater than 50 percentage points, and maintains 81.7% R@1 on the extended configuration while establishing state-of-the-art results in three conventional benchmarks (MSR-VTT, MSVD, and VATEX).