A LoRA is Worth a Thousand Pictures

TL;DR

This paper demonstrates that LoRA weights can effectively describe artistic styles, enabling style clustering and retrieval without image generation or training set knowledge, advancing style analysis in diffusion models.

Contribution

It reveals that LoRA weights alone can serve as style descriptors, outperforming traditional features in clustering and retrieval tasks, and discusses future applications like zero-shot fine-tuning.

Findings

LoRA weights outperform traditional features in style clustering.

LoRA-based embeddings show structural similarity to image-based embeddings.

Approach enables accurate style retrieval without training image knowledge.

Abstract

Recent advances in diffusion models and parameter-efficient fine-tuning (PEFT) have made text-to-image generation and customization widely accessible, with Low Rank Adaptation (LoRA) able to replicate an artist's style or subject using minimal data and computation. In this paper, we examine the relationship between LoRA weights and artistic styles, demonstrating that LoRA weights alone can serve as an effective descriptor of style, without the need for additional image generation or knowledge of the original training set. Our findings show that LoRA weights yield better performance in clustering of artistic styles compared to traditional pre-trained features, such as CLIP and DINO, with strong structural similarities between LoRA-based and conventional image-based embeddings observed both qualitatively and quantitatively. We identify various retrieval scenarios for the growing collection of customized models and show that our approach enables more accurate retrieval in real-world settings where knowledge of the training images is unavailable and additional generation is required. We conclude with a discussion on potential future applications, such as zero-shot LoRA fine-tuning and model attribution.