What Matters for Ad-hoc Video Search? A Large-scale Evaluation on TRECVID

TL;DR

This paper presents a large-scale, systematic evaluation of various components in ad-hoc video search solutions using TRECVID data, providing insights into what factors most influence performance.

Contribution

It introduces a comprehensive evaluation framework that compares different models, features, and training data combinations to identify key factors affecting AVS performance.

Findings

Certain visual features significantly improve search accuracy.

Cross-modal matching models vary in effectiveness.

Training data quality impacts overall system performance.

Abstract

For quantifying progress in Ad-hoc Video Search (AVS), the annual TRECVID AVS task is an important international evaluation. Solutions submitted by the task participants vary in terms of their choices of cross-modal matching models, visual features and training data. As such, what one may conclude from the evaluation is at a high level that is insufficient to reveal the influence of the individual components. In order to bridge the gap between the current solution-level comparison and the desired component-wise comparison, we propose in this paper a large-scale and systematic evaluation on TRECVID. By selected combinations of state-of-the-art matching models, visual features and (pre-)training data, we construct a set of 25 different solutions and evaluate them on the TRECVID AVS tasks 2016--2020. The presented evaluation helps answer the key question of what matters for AVS. The resultant observations and learned lessons are also instructive for developing novel AVS solutions.