Video2LoRA: Unified Semantic-Controlled Video Generation via Per-Reference-Video LoRA

TL;DR

Video2LoRA introduces a scalable, reference-video conditioned framework for semantic-controlled video generation, enabling flexible, high-quality, zero-shot generation without per-condition training.

Contribution

It presents a lightweight hypernetwork-based LoRA approach that allows semantic video generation conditioned on reference videos, improving flexibility and efficiency.

Findings

Generates semantically aligned videos with diverse conditions.

Model weights are less than 150MB, enabling efficient deployment.

Achieves strong zero-shot generalization to unseen semantics.

Abstract

Achieving semantic alignment across diverse video generation conditions remains a significant challenge. Methods that rely on explicit structural guidance often enforce rigid spatial constraints that limit semantic flexibility, whereas models tailored for individual control types lack interoperability and adaptability. These design bottlenecks hinder progress toward flexible and efficient semantic video generation. To address this, we propose Video2LoRA, a scalable and generalizable framework for semantic-controlled video generation that conditions on a reference video. Video2LoRA employs a lightweight hypernetwork to predict personalized LoRA weights for each semantic input, which are combined with auxiliary matrices to form adaptive LoRA modules integrated into a frozen diffusion backbone. This design enables the model to generate videos consistent with the reference semantics while preserving key style and content variations, eliminating the need for any per-condition training. Notably, the final model weights less than 150MB, making it highly efficient for storage and deployment. Video2LoRA achieves coherent, semantically aligned generation across diverse conditions and exhibits strong zero-shot generalization to unseen semantics.