DICE: Disentangling Artist Style from Content via Contrastive Subspace Decomposition in Diffusion Models

TL;DR

DICE is a training-free, contrastive subspace decomposition method that effectively erases artist style from diffusion models on-the-fly, balancing style removal and content preservation with minimal overhead.

Contribution

DICE introduces a novel contrastive triplet-based approach and eigenvalue formalization for style-content disentanglement without requiring explicit style samples or costly weight editing.

Findings

Achieves superior style erasure while preserving content integrity.

Operates with only 3 seconds overhead for style disentanglement.

Outperforms existing methods in style removal effectiveness.

Abstract

The recent proliferation of diffusion models has made style mimicry effortless, enabling users to imitate unique artistic styles without authorization. In deployed platforms, this raises copyright and intellectual-property risks and calls for reliable protection. However, existing countermeasures either require costly weight editing as new styles emerge or rely on an explicitly specified editing style, limiting their practicality for deployment-side safety. To address this challenge, we propose DICE (Disentanglement of artist Style from Content via Contrastive Subspace Decomposition), a training-free framework for on-the-fly artist style erasure. Unlike style editing that require an explicitly specified replacement style, DICE performs style purification, removing the artist's characteristics while preserving the user-intended content. Our core insight is that a model cannot truly comprehend the artist style from a single text or image alone. Consequently, we abandon the traditional paradigm of identifying style from isolated samples. Instead, we construct contrastive triplets to compel the model to distinguish between style and non-style features in the latent space. By formalizing this disentanglement process as a solvable generalized eigenvalue problem, we achieve precise identification of the style subspace. Furthermore, we introduce an Adaptive Attention Decoupling Editing strategy dynamically assesses the style concentration of each token and performs differential suppression and content enhancement on the QKV vectors. Extensive experiments demonstrate that DICE achieves a superior balance between the thoroughness of style erasure and the preservation of content integrity. DICE introduces an additional overhead of only 3 seconds to disentangle style, providing a practical and efficient technique for curbing style mimicry.