DreamVE: Unified Instruction-based Image and Video Editing
Bin Xia, Jiyang Liu, Yuechen Zhang, Bohao Peng, Ruihang Chu, Yitong Wang, Xinglong Wu, Bei Yu, and Jiaya Jia

TL;DR
DreamVE is a unified model for instruction-based image and video editing that leverages a two-stage training strategy and diverse data synthesis pipelines to improve editing capabilities and generalization.
Contribution
The paper introduces a novel two-stage training approach and comprehensive data synthesis methods for unified image and video editing models.
Findings
Pretrained on collage-based data, DreamVE achieves strong editing performance.
Fine-tuning with generative model-based data improves attribute editing.
Unified model enhances efficiency and transferability.
Abstract
Instruction-based editing holds vast potential due to its simple and efficient interactive editing format. However, instruction-based editing, particularly for video, has been constrained by limited training data, hindering its practical application. To this end, we introduce DreamVE, a unified model for instruction-based image and video editing. Specifically, We propose a two-stage training strategy: first image editing, then video editing. This offers two main benefits: (1) Image data scales more easily, and models are more efficient to train, providing useful priors for faster and better video editing training. (2) Unifying image and video generation is natural and aligns with current trends. Moreover, we present comprehensive training data synthesis pipelines, including collage-based and generative model-based data synthesis. The collage-based data synthesis combines foreground…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsVideo Analysis and Summarization · Advanced Vision and Imaging · Generative Adversarial Networks and Image Synthesis
