Beyond Closed-Pool Video Retrieval: A Benchmark and Agent Framework for Real-World Video Search and Moment Localization

TL;DR

This paper introduces RVMS-Bench, a new benchmark with diverse, real-world video samples and a hierarchical description framework, along with RACLO, an agentic reasoning framework, to improve video search and localization based on fuzzy, multi-dimensional memories.

Contribution

It presents RVMS-Bench for evaluating real-world video memory search and RACLO, an agentic framework employing abductive reasoning for more realistic video retrieval and localization.

Findings

Existing models underperform on real-world fuzzy memory tasks.

RVMS-Bench covers diverse categories and durations from open-web videos.

RACLO improves search accuracy by mimicking human recall processes.

Abstract

Traditional video retrieval benchmarks focus on matching precise descriptions to closed video pools, failing to reflect real-world searches characterized by fuzzy, multi-dimensional memories on the open web. We present \textbf{RVMS-Bench}, a comprehensive system for evaluating real-world video memory search. It consists of \textbf{1,440 samples} spanning \textbf{20 diverse categories} and \textbf{four duration groups}, sourced from \textbf{real-world open-web videos}. RVMS-Bench utilizes a hierarchical description framework encompassing \textbf{Global Impression, Key Moment, Temporal Context, and Auditory Memory} to mimic realistic multi-dimensional search cues, with all samples strictly verified via a human-in-the-loop protocol. We further propose \textbf{RACLO}, an agentic framework that employs abductive reasoning to simulate the human ``Recall-Search-Verify'' cognitive process, effectively addressing the challenge of searching for videos via fuzzy memories in the real world. Experiments reveal that existing MLLMs still demonstrate insufficient capabilities in real-world Video Retrieval and Moment Localization based on fuzzy memories. We believe this work will facilitate the advancement of video retrieval robustness in real-world unstructured scenarios.