Not All Pairs are Equal: Hierarchical Learning for Average-Precision-Oriented Video Retrieval

TL;DR

This paper introduces HAP-VR, a hierarchical learning framework that improves video retrieval by directly optimizing for Average Precision, addressing limitations of pair-wise losses and frame matching noise.

Contribution

It proposes novel similarity measures and loss functions tailored for AP optimization, along with a frame similarity constraint to reduce noise in video retrieval.

Findings

HAP-VR outperforms existing methods on benchmark datasets.

The proposed loss functions better align training with evaluation metrics.

Frame similarity constraints improve AP estimation accuracy.

Abstract

The rapid growth of online video resources has significantly promoted the development of video retrieval methods. As a standard evaluation metric for video retrieval, Average Precision (AP) assesses the overall rankings of relevant videos at the top list, making the predicted scores a reliable reference for users. However, recent video retrieval methods utilize pair-wise losses that treat all sample pairs equally, leading to an evident gap between the training objective and evaluation metric. To effectively bridge this gap, in this work, we aim to address two primary challenges: a) The current similarity measure and AP-based loss are suboptimal for video retrieval; b) The noticeable noise from frame-to-frame matching introduces ambiguity in estimating the AP loss. In response to these challenges, we propose the Hierarchical learning framework for Average-Precision-oriented Video Retrieval (HAP-VR). For the former challenge, we develop the TopK-Chamfer Similarity and QuadLinear-AP loss to measure and optimize video-level similarities in terms of AP. For the latter challenge, we suggest constraining the frame-level similarities to achieve an accurate AP loss estimation. Experimental results present that HAP-VR outperforms existing methods on several benchmark datasets, providing a feasible solution for video retrieval tasks and thus offering potential benefits for the multi-media application.