TriLoRA: Integrating SVD for Advanced Style Personalization in Text-to-Image Generation

TL;DR

TriLoRA introduces a novel integration of SVD into the LoRA framework to improve fine-tuning efficiency, stability, and feature capture in text-to-image models, enhancing creative flexibility and generalization without increasing resource demands.

Contribution

The paper presents a new method combining SVD with LoRA for better fine-tuning of image generation models, reducing overfitting and improving output quality and stability.

Findings

Enhanced model generalization and creative flexibility.

Significant improvement in output stability and feature accuracy.

Maintains efficiency under resource constraints.

Abstract

As deep learning technology continues to advance, image generation models, especially models like Stable Diffusion, are finding increasingly widespread application in visual arts creation. However, these models often face challenges such as overfitting, lack of stability in generated results, and difficulties in accurately capturing the features desired by creators during the fine-tuning process. In response to these challenges, we propose an innovative method that integrates Singular Value Decomposition (SVD) into the Low-Rank Adaptation (LoRA) parameter update strategy, aimed at enhancing the fine-tuning efficiency and output quality of image generation models. By incorporating SVD within the LoRA framework, our method not only effectively reduces the risk of overfitting but also enhances the stability of model outputs, and captures subtle, creator-desired feature adjustments more accurately. We evaluated our method on multiple datasets, and the results show that, compared to traditional fine-tuning methods, our approach significantly improves the model's generalization ability and creative flexibility while maintaining the quality of generation. Moreover, this method maintains LoRA's excellent performance under resource-constrained conditions, allowing for significant improvements in image generation quality without sacrificing the original efficiency and resource advantages.