MEVG: Multi-event Video Generation with Text-to-Video Models

TL;DR

This paper presents MEVG, a diffusion-based method for generating multi-event videos from multiple text inputs without fine-tuning, ensuring temporal coherence and semantic accuracy through novel diffusion processes and prompt generation.

Contribution

The paper introduces a new diffusion-based approach that generates multi-event videos from text without requiring large datasets or fine-tuning, using a last frame-aware diffusion process and a prompt generator.

Findings

Outperforms existing models in temporal coherence and semantic accuracy

Maintains global appearance across frames through iterative latent updates

Effective in generating videos with multiple distinct events

Abstract

We introduce a novel diffusion-based video generation method, generating a video showing multiple events given multiple individual sentences from the user. Our method does not require a large-scale video dataset since our method uses a pre-trained diffusion-based text-to-video generative model without a fine-tuning process. Specifically, we propose a last frame-aware diffusion process to preserve visual coherence between consecutive videos where each video consists of different events by initializing the latent and simultaneously adjusting noise in the latent to enhance the motion dynamic in a generated video. Furthermore, we find that the iterative update of latent vectors by referring to all the preceding frames maintains the global appearance across the frames in a video clip. To handle dynamic text input for video generation, we utilize a novel prompt generator that transfers course text messages from the user into the multiple optimal prompts for the text-to-video diffusion model. Extensive experiments and user studies show that our proposed method is superior to other video-generative models in terms of temporal coherency of content and semantics. Video examples are available on our project page: https://kuai-lab.github.io/eccv2024mevg.