Hollowed Net for On-Device Personalization of Text-to-Image Diffusion Models

TL;DR

This paper introduces Hollowed Net, a memory-efficient diffusion model fine-tuning method for on-device personalization, enabling high-quality subject-driven image generation with minimal resource use.

Contribution

Hollowed Net modifies the diffusion U-Net architecture to reduce memory during fine-tuning, facilitating on-device personalization without extra inference overhead.

Findings

Reduces GPU memory for training to inference levels

Maintains or improves personalization quality

Enables on-device subject-driven image generation

Abstract

Recent advancements in text-to-image diffusion models have enabled the personalization of these models to generate custom images from textual prompts. This paper presents an efficient LoRA-based personalization approach for on-device subject-driven generation, where pre-trained diffusion models are fine-tuned with user-specific data on resource-constrained devices. Our method, termed Hollowed Net, enhances memory efficiency during fine-tuning by modifying the architecture of a diffusion U-Net to temporarily remove a fraction of its deep layers, creating a hollowed structure. This approach directly addresses on-device memory constraints and substantially reduces GPU memory requirements for training, in contrast to previous methods that primarily focus on minimizing training steps and reducing the number of parameters to update. Additionally, the personalized Hollowed Net can be transferred back into the original U-Net, enabling inference without additional memory overhead. Quantitative and qualitative analyses demonstrate that our approach not only reduces training memory to levels as low as those required for inference but also maintains or improves personalization performance compared to existing methods.