StoryMem: Multi-shot Long Video Storytelling with Memory

TL;DR

StoryMem introduces a memory-augmented diffusion framework for long-form video storytelling, enabling coherent, high-quality multi-shot videos by dynamically maintaining visual memory and fine-tuning pre-trained models.

Contribution

The paper presents a novel Memory-to-Video design that transforms single-shot diffusion models into multi-shot storytellers with explicit visual memory and a new benchmark for evaluation.

Findings

Achieves superior cross-shot consistency compared to previous methods.

Maintains high aesthetic quality and adherence to prompts.

Supports smooth shot transitions and customized storytelling.

Abstract

Visual storytelling requires generating multi-shot videos with cinematic quality and long-range consistency. Inspired by human memory, we propose StoryMem, a paradigm that reformulates long-form video storytelling as iterative shot synthesis conditioned on explicit visual memory, transforming pre-trained single-shot video diffusion models into multi-shot storytellers. This is achieved by a novel Memory-to-Video (M2V) design, which maintains a compact and dynamically updated memory bank of keyframes from historical generated shots. The stored memory is then injected into single-shot video diffusion models via latent concatenation and negative RoPE shifts with only LoRA fine-tuning. A semantic keyframe selection strategy, together with aesthetic preference filtering, further ensures informative and stable memory throughout generation. Moreover, the proposed framework naturally accommodates smooth shot transitions and customized story generation applications. To facilitate evaluation, we introduce ST-Bench, a diverse benchmark for multi-shot video storytelling. Extensive experiments demonstrate that StoryMem achieves superior cross-shot consistency over previous methods while preserving high aesthetic quality and prompt adherence, marking a significant step toward coherent minute-long video storytelling.