Enhancing Privacy in ControlNet and Stable Diffusion via Split Learning

TL;DR

This paper introduces a novel distributed training framework for ControlNet and diffusion models that enhances user data privacy, counters existing threats, and maintains high-quality image generation.

Contribution

We propose a new distributed learning structure that avoids server gradient sharing, along with privacy-preserving techniques tailored for diffusion-based image generation.

Findings

The new framework effectively prevents most existing privacy attacks.

Our methods maintain image quality while enhancing privacy and training efficiency.

Experimental results confirm the robustness and practicality of the proposed approach.

Abstract

With the emerging trend of large generative models, ControlNet is introduced to enable users to fine-tune pre-trained models with their own data for various use cases. A natural question arises: how can we train ControlNet models while ensuring users' data privacy across distributed devices? Exploring different distributed training schemes, we find conventional federated learning and split learning unsuitable. Instead, we propose a new distributed learning structure that eliminates the need for the server to send gradients back. Through a comprehensive evaluation of existing threats, we discover that in the context of training ControlNet with split learning, most existing attacks are ineffective, except for two mentioned in previous literature. To counter these threats, we leverage the properties of diffusion models and design a new timestep sampling policy during forward processes. We further propose a privacy-preserving activation function and a method to prevent private text prompts from leaving clients, tailored for image generation with diffusion models. Our experimental results demonstrate that our algorithms and systems greatly enhance the efficiency of distributed training for ControlNet while ensuring users' data privacy without compromising image generation quality.