AutoVFX: Physically Realistic Video Editing from Natural Language Instructions
Hao-Yu Hsu, Zhi-Hao Lin, Albert Zhai, Hongchi Xia, Shenlong Wang

TL;DR
AutoVFX is a novel framework that enables realistic, physically grounded video editing from a single video using natural language instructions, making VFX creation more accessible and versatile.
Contribution
The paper introduces AutoVFX, integrating neural scene modeling, LLM-based code generation, and physical simulation for natural language-controlled video editing.
Findings
Outperforms competing methods in generative quality
Achieves high instruction alignment and editing versatility
Ensures physical plausibility in edited videos
Abstract
Modern visual effects (VFX) software has made it possible for skilled artists to create imagery of virtually anything. However, the creation process remains laborious, complex, and largely inaccessible to everyday users. In this work, we present AutoVFX, a framework that automatically creates realistic and dynamic VFX videos from a single video and natural language instructions. By carefully integrating neural scene modeling, LLM-based code generation, and physical simulation, AutoVFX is able to provide physically-grounded, photorealistic editing effects that can be controlled directly using natural language instructions. We conduct extensive experiments to validate AutoVFX's efficacy across a diverse spectrum of videos and instructions. Quantitative and qualitative results suggest that AutoVFX outperforms all competing methods by a large margin in generative quality, instruction…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Multimodal Machine Learning Applications · Video Analysis and Summarization
