Exploring Data-Free LoRA Transferability for Video Diffusion Models

TL;DR

This paper introduces CASA, a data-free framework that improves the transferability of LoRA adapters in video diffusion models by resolving spectral conflicts, leading to artifact reduction and better model adaptation.

Contribution

The paper identifies spectral interference as the cause of LoRA transfer issues and proposes CASA, a novel spectral arbitration method to enhance transferability without additional data.

Findings

CASA effectively mitigates artifacts in video diffusion models.

It successfully restores LoRA alignment and transferability.

Experiments show improved performance over existing methods.

Abstract

Video diffusion models leveraging step distillation or causal distillation have achieved remarkable performance. However, adapting existing LoRAs to these variants remains a critical challenge due to weight space mismatches. We observe that direct application leads to style degradation and structural collapse, yet the underlying mechanisms remain poorly understood. To fill this gap, we delve into the weight space and identify that the incompatibility stems from spectral interference within shared functional clusters defined over singular subspaces. Specifically, our analysis reveals that while both paradigms respect spectral rigidity, they establish conflicting routing pathways that clash through constructive overload or destructive cancellation. To address this issue, we propose Cluster-Aware Spectral Arbitration (CASA), a data-free framework that dynamically arbitrates between safeguarding the target's manifold and restoring LoRA alignment based on spectral density. Extensive experiments demonstrate that CASA effectively mitigates artifacts and revives LoRA functionality. Our code is available at https://github.com/Noahwangyuchen/CASA