Uncertainty-Aware and Decoder-Aligned Learning for Video Summarization

TL;DR

This paper introduces VASTSum, a novel video summarization framework that models uncertainty and aligns with decoding procedures, improving robustness and efficiency over existing methods.

Contribution

VASTSum is the first single-pass, uncertainty-aware, decoder-aligned learning framework for video summarization that explicitly models annotation subjectivity and stabilizes summary selection.

Findings

Achieves consistent Kendall and Spearman correlations on SumMe and TVSum.

Demonstrates improved robustness to annotation disagreement.

Maintains efficient single-forward inference.

Abstract

Video summarization aims to produce a compact representation of a long video by selecting a subset of temporally important segments that best reflect human preferences. This task is inherently difficult due to strong annotation subjectivity and the reliance on discrete decoding procedures, such as temporal segmentation and knapsack-based selection, during evaluation. Most existing approaches either learn deterministic importance scores that overlook these characteristics or adopt complex generative models that increase training and inference cost. In this paper, we propose VASTSum, an uncertainty-aware and decoder-aligned learning framework for video summarization that addresses both challenges within a single-pass model. The proposed method predicts probabilistic frame-level importance scores using a variational formulation, enabling explicit modeling of uncertainty arising from multi-annotator supervision. To account for subjectivity, particularly under binary annotations, we employ a supervision strategy that encourages alignment with plausible human annotation modes rather than enforcing a single consensus target. Furthermore, we introduce a decoder-aligned regularization that promotes stability of knapsack-based summary selection, reducing sensitivity to small perturbations in predicted scores. We evaluate the proposed framework on the SumMe and TVSum benchmarks using standard rank-based metrics. Experimental results show consistent and competitive Kendall and Spearman correlations across multiple data splits, demonstrating improved robustness under annotation disagreement while maintaining efficient single-forward inference. These results indicate that explicitly modeling uncertainty and aligning learning objectives with the decoding stage provide a principled alternative to both deterministic and diffusion-based video summarization methods.