OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer

TL;DR

OmniShotCut introduces a Transformer-based approach for shot boundary detection that jointly predicts shot ranges and relations, utilizing synthetic data and a new benchmark for improved accuracy and interpretability.

Contribution

It formulates shot boundary detection as structured relational prediction using a shot query Transformer and creates synthetic data and a comprehensive benchmark.

Findings

Effective in detecting shot boundaries with high accuracy.

Generalizes well across diverse video domains.

Outperforms existing methods on new benchmark.

Abstract

Shot Boundary Detection (SBD) aims to automatically identify shot changes and divide a video into coherent shots. While SBD was widely studied in the literature, existing methods often produce non-interpretable boundaries on transitions, miss subtle yet harmful discontinuities, and rely on noisy, low-diversity annotations and outdated benchmarks. To alleviate these limitations, we propose OmniShotCut to formulate SBD as structured relational prediction, jointly estimating shot ranges with intra-shot relations and inter-shot relations, by a shot query-based dense video Transformer. To avoid imprecise manual labeling, we adopt a fully synthetic transition synthesis pipeline that automatically reproduces major transition families with precise boundaries and parameterized variants. We also introduce OmniShotCutBench, a modern wide-domain benchmark enabling holistic and diagnostic evaluation. Experiments on the benchmarks demonstrate the effectiveness and generality of our method.