TL;DR
Aurora is a unified video editing framework that uses a tool-augmented vision-language model to interpret user requests and generate detailed edit plans, improving flexibility and robustness in video editing tasks.
Contribution
The paper introduces Aurora, combining a VLM agent with a diffusion transformer to handle underspecified editing requests and demonstrates its effectiveness through new benchmarks and transferability tests.
Findings
Aurora outperforms instruction-only baselines on AgentEdit-Bench.
The VLM agent effectively resolves underspecification in user requests.
Aurora's approach transfers well to other video editing models.
Abstract
Recent video editing models have converged on a unified conditioning design: a single diffusion transformer jointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit. We present Aurora, an agentic video editing framework that pairs a tool-augmented vision-language model (VLM) agent with a unified video diffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer's conditioning channels, thereby resolving textual and visual underspecification before generation. We train the VLM agent with supervised data for complete edit planning and reference-image selection,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
