Modality-Aware Shot Relating and Comparing for Video Scene Detection

TL;DR

This paper introduces MASRC, a novel modality-aware approach for video scene detection that leverages visual entity and place semantics to improve shot relation modeling and scene boundary identification.

Contribution

The paper proposes a modality-aware shot relating and comparing method that explicitly models long-term and short-term shot correlations using multi-modal semantics, enhancing scene detection accuracy.

Findings

MASRC outperforms existing methods on benchmark datasets.

Explicit modeling of multi-modal shot relations improves detection performance.

Long-term entity and short-term place semantics effectively distinguish scene boundaries.

Abstract

Video scene detection involves assessing whether each shot and its surroundings belong to the same scene. Achieving this requires meticulously correlating multi-modal cues, $\it{e.g.}$ visual entity and place modalities, among shots and comparing semantic changes around each shot. However, most methods treat multi-modal semantics equally and do not examine contextual differences between the two sides of a shot, leading to sub-optimal detection performance. In this paper, we propose the $\bf{M}$odality-$\bf{A}$ware $\bf{S}$hot $\bf{R}$elating and $\bf{C}$omparing approach (MASRC), which enables relating shots per their own characteristics of visual entity and place modalities, as well as comparing multi-shots similarities to have scene changes explicitly encoded. Specifically, to fully harness the potential of visual entity and place modalities in modeling shot relations, we mine long-term shot correlations from entity semantics while simultaneously revealing short-term shot correlations from place semantics. In this way, we can learn distinctive shot features that consolidate coherence within scenes and amplify distinguishability across scenes. Once equipped with distinctive shot features, we further encode the relations between preceding and succeeding shots of each target shot by similarity convolution, aiding in the identification of scene ending shots. We validate the broad applicability of the proposed components in MASRC. Extensive experimental results on public benchmark datasets demonstrate that the proposed MASRC significantly advances video scene detection.