MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images

TL;DR

MovieFactory is an innovative framework that automatically generates multi-scene, cinematic movies with synchronized sound from natural language inputs, utilizing large generative models for both images and audio.

Contribution

It introduces the first fully automated movie generation system that creates high-quality, multi-modality movies from simple text, surpassing previous soundless and single-scene methods.

Findings

Produces realistic, diverse, multi-scene movies with synchronized audio

Uses a two-stage process for video generation involving spatial finetuning and temporal learning

Demonstrates high-quality results with immersive visual and auditory experiences

Abstract

In this paper, we present MovieFactory, a powerful framework to generate cinematic-picture (3072$\times$1280), film-style (multi-scene), and multi-modality (sounding) movies on the demand of natural languages. As the first fully automated movie generation model to the best of our knowledge, our approach empowers users to create captivating movies with smooth transitions using simple text inputs, surpassing existing methods that produce soundless videos limited to a single scene of modest quality. To facilitate this distinctive functionality, we leverage ChatGPT to expand user-provided text into detailed sequential scripts for movie generation. Then we bring scripts to life visually and acoustically through vision generation and audio retrieval. To generate videos, we extend the capabilities of a pretrained text-to-image diffusion model through a two-stage process. Firstly, we employ spatial finetuning to bridge the gap between the pretrained image model and the new video dataset. Subsequently, we introduce temporal learning to capture object motion. In terms of audio, we leverage sophisticated retrieval models to select and align audio elements that correspond to the plot and visual content of the movie. Extensive experiments demonstrate that our MovieFactory produces movies with realistic visuals, diverse scenes, and seamlessly fitting audio, offering users a novel and immersive experience. Generated samples can be found in YouTube or Bilibili (1080P).