UnHype: CLIP-Guided Hypernetworks for Dynamic LoRA Unlearning

TL;DR

UnHype introduces a hypernetwork-based framework that enhances the efficiency and adaptability of concept unlearning in large-scale diffusion models, enabling more precise and scalable removal of specific concepts during image generation.

Contribution

The paper presents a novel hypernetwork approach integrated with LoRA for improved, scalable, and context-aware concept unlearning in diffusion models.

Findings

Effective object, celebrity, and content removal demonstrated.

Stable training behavior across multiple tasks.

Enhanced scalability and concept control during inference.

Abstract

Recent advances in large-scale diffusion models have intensified concerns about their potential misuse, particularly in generating realistic yet harmful or socially disruptive content. This challenge has spurred growing interest in effective machine unlearning, the process of selectively removing specific knowledge or concepts from a model without compromising its overall generative capabilities. Among various approaches, Low-Rank Adaptation (LoRA) has emerged as an effective and efficient method for fine-tuning models toward targeted unlearning. However, LoRA-based methods often exhibit limited adaptability to concept semantics and struggle to balance removing closely related concepts with maintaining generalization across broader meanings. Moreover, these methods face scalability challenges when multiple concepts must be erased simultaneously. To address these limitations, we introduce UnHype, a framework that incorporates hypernetworks into single- and multi-concept LoRA training. The proposed architecture can be directly plugged into Stable Diffusion as well as modern flow-based text-to-image models, where it demonstrates stable training behavior and effective concept control. During inference, the hypernetwork dynamically generates adaptive LoRA weights based on the CLIP embedding, enabling more context-aware, scalable unlearning. We evaluate UnHype across several challenging tasks, including object erasure, celebrity erasure, and explicit content removal, demonstrating its effectiveness and versatility. Repository: https://github.com/gmum/UnHype.