Implicit Style-Content Separation using B-LoRA

TL;DR

This paper introduces B-LoRA, a novel method leveraging Low-Rank Adaptation to implicitly separate style and content in images, enabling versatile stylization tasks without extensive retraining.

Contribution

The paper presents B-LoRA, a new approach that uses joint learning of LoRA weights in two specific blocks to achieve effective style-content separation in images.

Findings

Joint learning of two B-LoRAs outperforms independent training.

B-LoRA improves style manipulation and reduces overfitting.

The method enables diverse stylization tasks like style transfer and content mixing.

Abstract

Image stylization involves manipulating the visual appearance and texture (style) of an image while preserving its underlying objects, structures, and concepts (content). The separation of style and content is essential for manipulating the image's style independently from its content, ensuring a harmonious and visually pleasing result. Achieving this separation requires a deep understanding of both the visual and semantic characteristics of images, often necessitating the training of specialized models or employing heavy optimization. In this paper, we introduce B-LoRA, a method that leverages LoRA (Low-Rank Adaptation) to implicitly separate the style and content components of a single image, facilitating various image stylization tasks. By analyzing the architecture of SDXL combined with LoRA, we find that jointly learning the LoRA weights of two specific blocks (referred to as B-LoRAs) achieves style-content separation that cannot be achieved by training each B-LoRA independently. Consolidating the training into only two blocks and separating style and content allows for significantly improving style manipulation and overcoming overfitting issues often associated with model fine-tuning. Once trained, the two B-LoRAs can be used as independent components to allow various image stylization tasks, including image style transfer, text-based image stylization, consistent style generation, and style-content mixing.