Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks

TL;DR

This paper reveals that current text-to-video retrieval benchmarks are flawed due to false negatives, leading to underestimation of model performance, and proposes improved evaluation methods and new annotations.

Contribution

The authors identify false negatives in existing benchmarks, provide a large-scale annotated dataset, and recommend retiring current benchmarks in favor of more accurate evaluation methods.

Findings

Recomputed metrics show up to 25% higher recall for top models.

Benchmarks are nearing saturation for Recall@10.

Annotation costs can be reduced through sampling.

Abstract

Searching troves of videos with textual descriptions is a core multimodal retrieval task. Owing to the lack of a purpose-built dataset for text-to-video retrieval, video captioning datasets have been re-purposed to evaluate models by (1) treating captions as positive matches to their respective videos and (2) assuming all other videos to be negatives. However, this methodology leads to a fundamental flaw during evaluation: since captions are marked as relevant only to their original video, many alternate videos also match the caption, which introduces false-negative caption-video pairs. We show that when these false negatives are corrected, a recent state-of-the-art model gains 25\% recall points -- a difference that threatens the validity of the benchmark itself. To diagnose and mitigate this issue, we annotate and release 683K additional caption-video pairs. Using these, we recompute effectiveness scores for three models on two standard benchmarks (MSR-VTT and MSVD). We find that (1) the recomputed metrics are up to 25\% recall points higher for the best models, (2) these benchmarks are nearing saturation for Recall@10, (3) caption length (generality) is related to the number of positives, and (4) annotation costs can be mitigated through sampling. We recommend retiring these benchmarks in their current form, and we make recommendations for future text-to-video retrieval benchmarks.