Retrieval, Refinement, and Ranking for Text-to-Video Generation via Prompt Optimization and Test-Time Scaling

TL;DR

This paper introduces 3R, a prompt optimization framework for Text-to-Video generation that improves video quality and coherence without requiring model training, by leveraging retrieval, refinement, and ranking strategies.

Contribution

The paper presents a novel prompt optimization method for T2V models that enhances output quality using retrieval, preference optimization, and temporal interpolation, without additional model training.

Findings

Improves static fidelity of generated videos.

Enhances dynamic coherence and temporal consistency.

Effective across different T2V models without retraining.

Abstract

While large-scale datasets have driven significant progress in Text-to-Video (T2V) generative models, these models remain highly sensitive to input prompts, demonstrating that prompt design is critical to generation quality. Current methods for improving video output often fall short: they either depend on complex, post-editing models, risking the introduction of artifacts, or require expensive fine-tuning of the core generator, which severely limits both scalability and accessibility. In this work, we introduce 3R, a novel RAG based prompt optimization framework. 3R utilizes the power of current state-of-the-art T2V diffusion model and vision language model. It can be used with any T2V model without any kind of model training. The framework leverages three key strategies: RAG-based modifiers extraction for enriched contextual grounding, diffusion-based Preference Optimization for aligning outputs with human preferences, and temporal frame interpolation for producing temporally consistent visual contents. Together, these components enable more accurate, efficient, and contextually aligned text-to-video generation. Experimental results demonstrate the efficacy of 3R in enhancing the static fidelity and dynamic coherence of generated videos, underscoring the importance of optimizing user prompts.