RISE-T2V: Rephrasing and Injecting Semantics with LLM for Expansive Text-to-Video Generation

TL;DR

RISE-T2V introduces a unified approach combining prompt rephrasing and semantic extraction using LLMs to improve text-to-video generation quality and user intent alignment.

Contribution

The paper presents RISE-T2V, a novel framework that integrates prompt rephrasing with semantic feature extraction into a single step, enhancing T2V models' scalability and performance.

Findings

Significantly improves video quality with concise prompts.

Enhances alignment with user intent through prompt rephrasing.

Applicable to various pre-trained LLMs and diffusion models.

Abstract

Most text-to-video(T2V) diffusion models depend on pre-trained text encoders for semantic alignment, yet they often fail to maintain video quality when provided with concise prompts rather than well-designed ones. The primary issue lies in their limited textual semantics understanding. Moreover, these text encoders cannot rephrase prompts online to better align with user intentions, which limits both the scalability and usability of the models, To address these challenges, we introduce RISE-T2V, which uniquely integrates the processes of prompt rephrasing and semantic feature extraction into a single and seamless step instead of two separate steps. RISE-T2V is universal and can be applied to various pre-trained LLMs and video diffusion models(VDMs), significantly enhancing their capabilities for T2V tasks. We propose an innovative module called the Rephrasing Adapter, enabling diffusion models to utilize text hidden states during the next token prediction of the LLM as a condition for video generation. By employing a Rephrasing Adapter, the video generation model can implicitly rephrase basic prompts into more comprehensive representations that better match the user's intent. Furthermore, we leverage the powerful capabilities of LLMs to enable video generation models to accomplish a broader range of T2V tasks. Extensive experiments demonstrate that RISE-T2V is a versatile framework applicable to different video diffusion model architectures, significantly enhancing the ability of T2V models to generate high-quality videos that align with user intent. Visual results are available on the webpage at https://rise-t2v.github.io.