ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning

TL;DR

ViSiL introduces a novel CNN-based architecture for fine-grained spatio-temporal video similarity learning, capturing detailed intra- and inter-frame relations to improve video retrieval accuracy.

Contribution

The paper presents a new method that computes video similarity from frame-level relations without aggregating features prematurely, enhancing retrieval performance.

Findings

Achieves large improvements over state-of-the-art on five benchmark datasets.

Effectively captures temporal similarity patterns between matching frame sequences.

Demonstrates robustness across four different video retrieval tasks.

Abstract

In this paper we introduce ViSiL, a Video Similarity Learning architecture that considers fine-grained Spatio-Temporal relations between pairs of videos -- such relations are typically lost in previous video retrieval approaches that embed the whole frame or even the whole video into a vector descriptor before the similarity estimation. By contrast, our Convolutional Neural Network (CNN)-based approach is trained to calculate video-to-video similarity from refined frame-to-frame similarity matrices, so as to consider both intra- and inter-frame relations. In the proposed method, pairwise frame similarity is estimated by applying Tensor Dot (TD) followed by Chamfer Similarity (CS) on regional CNN frame features - this avoids feature aggregation before the similarity calculation between frames. Subsequently, the similarity matrix between all video frames is fed to a four-layer CNN, and then summarized using Chamfer Similarity (CS) into a video-to-video similarity score -- this avoids feature aggregation before the similarity calculation between videos and captures the temporal similarity patterns between matching frame sequences. We train the proposed network using a triplet loss scheme and evaluate it on five public benchmark datasets on four different video retrieval problems where we demonstrate large improvements in comparison to the state of the art. The implementation of ViSiL is publicly available.