TRIMMER: A New Paradigm for Video Summarization through Self-Supervised Reinforcement Learning

TL;DR

TRIMMER is a self-supervised reinforcement learning framework that improves video summarization by capturing temporal dynamics and semantic diversity efficiently, outperforming existing unsupervised methods.

Contribution

It introduces a novel entropy-based reward mechanism and a two-stage learning process for scalable, domain-agnostic video summarization.

Findings

Achieves state-of-the-art results among unsupervised methods

Maintains competitive performance with supervised approaches

Enhances efficiency through direct reward computation over frames

Abstract

The rapid growth of video content across domains such as surveillance, education, and social media has made efficient content understanding increasingly critical. Video summarization addresses this challenge by generating concise yet semantically meaningful representations, but existing approaches often rely on expensive manual annotations, struggle to generalize across domains, and incur significant computational costs due to complex architectures. Moreover, unsupervised and weakly supervised methods typically underperform compared to supervised counterparts in capturing long-range temporal dependencies and semantic structure. In this work, we propose TRIMMER (Temporal Relative Information Maximization for Multi-objective Efficient Reinforcement), a novel self-supervised reinforcement learning framework for video summarization. TRIMMER operates in two stages: it first learns robust representations via self-supervised learning and then performs spatio-temporal decision making through reinforcement learning guided by information-theoretic reward functions. Unlike prior approaches that rely on similarity-based objectives, our method introduces entropy-based metrics to capture higher-order temporal dynamics and semantic diversity, while computing rewards directly over selected frame indices to improve computational efficiency. Extensive experiments on standard benchmarks demonstrate that TRIMMER achieves state-of-the-art performance among unsupervised and self-supervised methods, while remaining competitive with leading supervised approaches, highlighting its effectiveness for scalable and generalizable video summarization.