DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing

TL;DR

The paper introduces DIRECT, a hierarchical multi-agent framework for automated video mashup creation that improves visual and auditory coherence, outperforming existing methods in quality and alignment.

Contribution

It formulates video mashup as a Multimodal Coherency Satisfaction Problem and proposes a novel hierarchical multi-agent approach with a new benchmark dataset.

Findings

DIRECT outperforms state-of-the-art baselines in objective metrics.

Extensive experiments show improved visual continuity and auditory alignment.

Human evaluations favor DIRECT's mashup quality.

Abstract

Video mashup creation represents a complex video editing paradigm that recomposes existing footage to craft engaging audio-visual experiences, demanding intricate orchestration across semantic, visual, and auditory dimensions and multiple levels. However, existing automated editing frameworks often overlook the cross-level multimodal orchestration to achieve professional-grade fluidity, resulting in disjointed sequences with abrupt visual transitions and musical misalignment. To address this, we formulate video mashup creation as a Multimodal Coherency Satisfaction Problem (MMCSP) and propose the DIRECT framework. Simulating a professional production pipeline, our hierarchical multi-agent framework decomposes the challenge into three cascade levels: the Screenwriter for source-aware global structural anchoring, the Director for instantiating adaptive editing intent and guidance, and the Editor for intent-guided shot sequence editing with fine-grained optimization. We further introduce Mashup-Bench, a comprehensive benchmark with tailored metrics for visual continuity and auditory alignment. Extensive experiments demonstrate that DIRECT significantly outperforms state-of-the-art baselines in both objective metrics and human subjective evaluation. Project page and code: https://github.com/AK-DREAM/DIRECT