AutoDirector: Online Auto-scheduling Agents for Multi-sensory Composition

TL;DR

AutoDirector is an interactive AI framework that automates online scheduling and multi-sensory composition, enabling efficient, user-guided creation of complex multi-sensory films with diverse elements.

Contribution

It introduces a novel online scheduling and interactive framework for multi-sensory film production, addressing dependency management and iterative user feedback.

Findings

Supports long shots, special effects, music, dubbing, lip-syncing

Improves efficiency of multi-sensory film production

Demonstrates AI-human collaboration in film directing

Abstract

With the advancement of generative models, the synthesis of different sensory elements such as music, visuals, and speech has achieved significant realism. However, the approach to generate multi-sensory outputs has not been fully explored, limiting the application on high-value scenarios such as of directing a film. Developing a movie director agent faces two major challenges: (1) Lack of parallelism and online scheduling with production steps: In the production of multi-sensory films, there are complex dependencies between different sensory elements, and the production time for each element varies. (2) Diverse needs and clear communication demands with users: Users often cannot clearly express their needs until they see a draft, which requires human-computer interaction and iteration to continually adjust and optimize the film content based on user feedback. To address these issues, we introduce AutoDirector, an interactive multi-sensory composition framework that supports long shots, special effects, music scoring, dubbing, and lip-syncing. This framework improves the efficiency of multi-sensory film production through automatic scheduling and supports the modification and improvement of interactive tasks to meet user needs. AutoDirector not only expands the application scope of human-machine collaboration but also demonstrates the potential of AI in collaborating with humans in the role of a film director to complete multi-sensory films.