Compare and Select: Video Summarization with Multi-Agent Reinforcement Learning

TL;DR

This paper introduces CoSNet, a multi-agent reinforcement learning framework for video summarization that models user-like decision processes, effectively handling subjectivity and outperforming existing methods.

Contribution

It proposes a novel multi-agent reinforcement learning approach inspired by user behavior, combining comparison and selection networks for improved video summarization.

Findings

Outperforms state-of-the-art unsupervised methods

Surpasses most supervised methods with full rewards

Effective in modeling subjective user preferences

Abstract

Video summarization aims at generating concise video summaries from the lengthy videos, to achieve better user watching experience. Due to the subjectivity, purely supervised methods for video summarization may bring the inherent errors from the annotations. To solve the subjectivity problem, we study the general user summarization process. General users usually watch the whole video, compare interesting clips and select some clips to form a final summary. Inspired by the general user behaviours, we formulate the summarization process as multiple sequential decision-making processes, and propose Comparison-Selection Network (CoSNet) based on multi-agent reinforcement learning. Each agent focuses on a video clip and constantly changes its focus during the iterations, and the final focus clips of all agents form the summary. The comparison network provides the agent with the visual feature from clips and the chronological feature from the past round, while the selection network of the agent makes decisions on the change of its focus clip. The specially designed unsupervised reward and supervised reward together contribute to the policy advancement, each containing local and global parts. Extensive experiments on two benchmark datasets show that CoSNet outperforms state-of-the-art unsupervised methods with the unsupervised reward and surpasses most supervised methods with the complete reward.