FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts

TL;DR

FoleyDirector introduces a novel framework for fine-grained temporal control in video-to-audio generation, utilizing structured scripts and a new dataset to improve controllability without sacrificing audio quality.

Contribution

It presents FoleyDirector, the first system enabling precise temporal guidance in V2A, with new modules, datasets, and evaluation benchmarks for enhanced controllability.

Findings

Significantly improves temporal controllability in V2A.

Maintains high audio fidelity comparable to baseline models.

Enables seamless switching between generation modes.

Abstract

Recent Video-to-Audio (V2A) methods have achieved remarkable progress, enabling the synthesis of realistic, high-quality audio. However, they struggle with fine-grained temporal control in multi-event scenarios or when visual cues are insufficient, such as small regions, off-screen sounds, or occluded or partially visible objects. In this paper, we propose FoleyDirector, a framework that, for the first time, enables precise temporal guidance in DiT-based V2A generation while preserving the base model's audio quality and allowing seamless switching between V2A generation and temporally controlled synthesis. FoleyDirector introduces Structured Temporal Scripts (STS), a set of captions corresponding to short temporal segments, to provide richer temporal information. These features are integrated via the Script-Guided Temporal Fusion Module, which employs Temporal Script Attention to fuse STS features coherently. To handle complex multi-event scenarios, we further propose Bi-Frame Sound Synthesis, enabling parallel in-frame and out-of-frame audio generation and improving controllability. To support training and evaluation, we construct the DirectorSound dataset and introduce VGGSoundDirector and DirectorBench. Experiments demonstrate that FoleyDirector substantially enhances temporal controllability while maintaining high audio fidelity, empowering users to act as Foley directors and advancing V2A toward more expressive and controllable generation.