Learning Segment Similarity and Alignment in Large-Scale Content Based Video Retrieval

TL;DR

This paper introduces SSAN, an end-to-end trainable network for segment-level video retrieval that improves alignment accuracy and efficiency by using novel keyframe extraction and similarity detection modules.

Contribution

The paper proposes a novel end-to-end trainable network with two new modules, SKE and SPD, enhancing segment alignment accuracy and efficiency in large-scale video retrieval.

Findings

SSAN outperforms existing methods in alignment accuracy.

SKE reduces storage and computation with minimal accuracy loss.

SPD improves temporal localization efficiency.

Abstract

With the explosive growth of web videos in recent years, large-scale Content-Based Video Retrieval (CBVR) becomes increasingly essential in video filtering, recommendation, and copyright protection. Segment-level CBVR (S-CBVR) locates the start and end time of similar segments in finer granularity, which is beneficial for user browsing efficiency and infringement detection especially in long video scenarios. The challenge of S-CBVR task is how to achieve high temporal alignment accuracy with efficient computation and low storage consumption. In this paper, we propose a Segment Similarity and Alignment Network (SSAN) in dealing with the challenge which is firstly trained end-to-end in S-CBVR. SSAN is based on two newly proposed modules in video retrieval: (1) An efficient Self-supervised Keyframe Extraction (SKE) module to reduce redundant frame features, (2) A robust Similarity Pattern Detection (SPD) module for temporal alignment. In comparison with uniform frame extraction, SKE not only saves feature storage and search time, but also introduces comparable accuracy and limited extra computation time. In terms of temporal alignment, SPD localizes similar segments with higher accuracy and efficiency than existing deep learning methods. Furthermore, we jointly train SSAN with SKE and SPD and achieve an end-to-end improvement. Meanwhile, the two key modules SKE and SPD can also be effectively inserted into other video retrieval pipelines and gain considerable performance improvements. Experimental results on public datasets show that SSAN can obtain higher alignment accuracy while saving storage and online query computational cost compared to existing methods.