PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction
Shangyu Chen, Zizheng Pan, Jianfei Cai, Dinh Phung
TL;DR
PaRa introduces a parameter rank reduction method for personalized text-to-image diffusion models, effectively balancing the learning of new concepts with preservation of text editability, and outperforming existing finetuning techniques in efficiency and accuracy.
Contribution
The paper presents a novel parameter rank reduction approach for T2I model personalization, improving efficiency and target alignment over existing methods like LoRA.
Findings
PaRa achieves better target image alignment than existing methods.
PaRa uses half the number of learnable parameters compared to LoRA.
Experiments show PaRa excels in single/multi-subject generation and image editing.
Abstract
Personalizing a large-scale pretrained Text-to-Image (T2I) diffusion model is challenging as it typically struggles to make an appropriate trade-off between its training data distribution and the target distribution, i.e., learning a novel concept with only a few target images to achieve personalization (aligning with the personalized target) while preserving text editability (aligning with diverse text prompts). In this paper, we propose PaRa, an effective and efficient Parameter Rank Reduction approach for T2I model personalization by explicitly controlling the rank of the diffusion model parameters to restrict its initial diverse generation space into a small and well-balanced target space. Our design is motivated by the fact that taming a T2I model toward a novel concept such as a specific art style implies a small generation space. To this end, by reducing the rank of model…
Peer Reviews
Decision·ICLR 2025 Spotlight
1. This paper is well-structured and easy to understand. 2. The idea of achieving T2I model personalization by controlling the rank of diffusion model parameters is highly innovative. Additionally, detailed explanations are provided for the introduced learnable low-rank parameters. 3. The experimental section is well-designed with sufficient data, demonstrating the effectiveness of PaRa in single/multi-subject generation and single-image editing, as well as its compatibility with other modules.
1. The methodology section includes extensive explanations of the mathematical principles behind PaRa but lacks an organized overview of the model’s framework. From the subsequent experimental section, it is evident that the approach also utilizes text embeddings, among other elements. While these are not the main focus of the methodology, they should be appropriately explained. Additionally, adding some visualizations in the methodology section would make the concepts more intuitive. 2. In the
1. Solving personalisation from matrix decomposition is a reasonable and good idea. 2. Combing rank with edibility, especially diversity of the generated image is proven effective in this paper. 3. The experiments are convincing in explaining the superiority of the proposed solution.
1. Matrix decomposition is proven to be effective in parameter efficient fine-tuning tasks, e.g. image editing. Although the difference between the proposed solution to Lora is clear. As both methods are based on matrix decomposition, it's not clear what are the foundational differences, and how the authors come up with the current solution. 2. SSIM is used in this paper for image diversity evaluation. I'm not quite sure whether SSIM is the best one, as SSIM focuses on pixel-level difference ins
1. The proposed Parameter Rank Reduction (PaRa) method is a creative solution to the challenge of T2I model personalization, offering a new perspective on controlling the generation space. 2. PaRa demonstrates significant parameter efficiency, requiring 2× fewer learnable parameters compared to existing methods like LoRA. 3. The paper provides extensive experimental results showing PaRa’s advantages in single/multi-subject generation and single-image editing and shows better results than LoRA. 4
1. The authors have not compared their method with SOTA such as LyCoris, DiffuseKronA, etc. Including these results would provide a better comparison of the method.
Videos
Taxonomy
TopicsImage Retrieval and Classification Techniques · AI in cancer detection · Video Analysis and Summarization
MethodsDiffusion