POS: A Prompts Optimization Suite for Augmenting Text-to-Video Generation
Shijie Ma, Huayi Xu, Mengjian Li, Weidong Geng, Yaxiong Wang, Meng, Wang

TL;DR
POS is a prompt optimization suite that enhances text-to-video generation by optimizing noise inputs and preserving semantic integrity in text prompts, leading to more stable and consistent video outputs without additional training.
Contribution
It introduces a training-free method to optimize noise and text prompts, improving video quality and consistency in diffusion-based text-to-video models.
Findings
Significant improvement in video quality and temporal consistency.
Effective noise approximation matching each textual input.
Semantic-preserving prompt rewriting enhances generation fidelity.
Abstract
This paper targets to enhance the diffusion-based text-to-video generation by improving the two input prompts, including the noise and the text. Accommodated with this goal, we propose POS, a training-free Prompt Optimization Suite to boost text-to-video models. POS is motivated by two observations: (1) Video generation shows instability in terms of noise. Given the same text, different noises lead to videos that differ significantly in terms of both frame quality and temporal consistency. This observation implies that there exists an optimal noise matched to each textual input; To capture the potential noise, we propose an optimal noise approximator to approach the potential optimal noise. Particularly, the optimal noise approximator initially searches a video that closely relates to the text prompt and then inverts it into the noise space to serve as an improved noise prompt for the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Video Analysis and Summarization · Music and Audio Processing
MethodsDiffusion
