Aurora: Unified Video Editing with a Tool-Using Agent

TL;DR

Aurora is a unified video editing framework that uses a tool-augmented vision-language model to interpret user requests and generate detailed edit plans, improving flexibility and robustness in video editing tasks.

Contribution

The paper introduces Aurora, combining a VLM agent with a diffusion transformer to handle underspecified editing requests and demonstrates its effectiveness through new benchmarks and transferability tests.

Findings

Aurora outperforms instruction-only baselines on AgentEdit-Bench.

The VLM agent effectively resolves underspecification in user requests.

Aurora's approach transfers well to other video editing models.

Abstract

Recent video editing models have converged on a unified conditioning design: a single diffusion transformer jointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit. We present Aurora, an agentic video editing framework that pairs a tool-augmented vision-language model (VLM) agent with a unified video diffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer's conditioning channels, thereby resolving textual and visual underspecification before generation. We train the VLM agent with supervised data for complete edit planning and reference-image selection, together with preference pairs for robust tool use and instruction refinement. We introduce AgentEdit-Bench to evaluate agent-enhanced video editing under textual and visual underspecification. Experiments on AgentEdit-Bench and two existing video editing benchmarks show that Aurora improves over instruction-only baselines and that the VLM agent transfers to compatible frozen video editing models. Project page: https://yeates.github.io/Aurora-Page