Self-Paced and Self-Corrective Masked Prediction for Movie Trailer Generation

TL;DR

This paper introduces SSMP, a novel self-paced, self-corrective masked prediction method using a Transformer encoder for automatic movie trailer generation, outperforming existing approaches through bi-directional contextual modeling and progressive self-correction.

Contribution

The paper proposes a new self-paced, self-corrective masked prediction framework for trailer generation, moving beyond the traditional selection-then-ranking paradigm with state-of-the-art results.

Findings

SSMP achieves superior trailer quality compared to existing methods.

The self-paced masking improves model adaptability and performance.

Progressive self-correction mimics human editing processes.

Abstract

As a challenging video editing task, movie trailer generation involves selecting and reorganizing movie shots to create engaging trailers. Currently, most existing automatic trailer generation methods employ a "selection-then-ranking" paradigm (i.e., first selecting key shots and then ranking them), which suffers from inevitable error propagation and limits the quality of the generated trailers. Beyond this paradigm, we propose a new self-paced and self-corrective masked prediction method called SSMP, which achieves state-of-the-art results in automatic trailer generation via bi-directional contextual modeling and progressive self-correction. In particular, SSMP trains a Transformer encoder that takes the movie shot sequences as prompts and generates corresponding trailer shot sequences accordingly. The model is trained via masked prediction, reconstructing each trailer shot sequence from its randomly masked counterpart. The mask ratio is self-paced, allowing the task difficulty to adapt to the model and thereby improving model performance. When generating a movie trailer, the model fills the shot positions with high confidence at each step and re-masks the remaining positions for the next prediction, forming a progressive self-correction mechanism that is analogous to how human editors work. Both quantitative results and user studies demonstrate the superiority of SSMP in comparison to existing automatic movie trailer generation methods. Demo is available at: https://github.com/Dixin-Lab/SSMP.