UnZipLoRA: Separating Content and Style from a Single Image

TL;DR

UnZipLoRA is a novel method that decomposes a single image into separate subject and style components using compatible Low-Rank Adaptations, enabling independent manipulation and recombination for creative applications.

Contribution

It introduces a simultaneous training approach for subject and style LoRAs from a single image, with techniques ensuring their compatibility and disentanglement.

Findings

Outperforms existing methods in preserving subject and style separation

Enables flexible manipulation and recombination of image elements

Validated through human studies and quantitative metrics

Abstract

This paper introduces UnZipLoRA, a method for decomposing an image into its constituent subject and style, represented as two distinct LoRAs (Low-Rank Adaptations). Unlike existing personalization techniques that focus on either subject or style in isolation, or require separate training sets for each, UnZipLoRA disentangles these elements from a single image by training both the LoRAs simultaneously. UnZipLoRA ensures that the resulting LoRAs are compatible, i.e., they can be seamlessly combined using direct addition. UnZipLoRA enables independent manipulation and recontextualization of subject and style, including generating variations of each, applying the extracted style to new subjects, and recombining them to reconstruct the original image or create novel variations. To address the challenge of subject and style entanglement, UnZipLoRA employs a novel prompt separation technique, as well as column and block separation strategies to accurately preserve the characteristics of subject and style, and ensure compatibility between the learned LoRAs. Evaluation with human studies and quantitative metrics demonstrates UnZipLoRA's effectiveness compared to other state-of-the-art methods, including DreamBooth-LoRA, Inspiration Tree, and B-LoRA.