Not All Parameters Matter: Masking Diffusion Models for Enhancing Generation Ability

TL;DR

This paper introduces MaskUNet, a parameter-efficient method that selectively masks diffusion model parameters to improve image generation quality, achieving state-of-the-art results with minimal additional parameters.

Contribution

The paper reveals the importance of specific U-Net parameters in diffusion models and proposes MaskUNet, a novel approach that enhances generation quality by zeroing out certain parameters, with effective fine-tuning strategies.

Findings

MaskUNet improves generation quality with negligible parameters.

Zero-shot inference achieves the best FID score on COCO.

Effective parameter masking enhances downstream task performance.

Abstract

The diffusion models, in early stages focus on constructing basic image structures, while the refined details, including local features and textures, are generated in later stages. Thus the same network layers are forced to learn both structural and textural information simultaneously, significantly differing from the traditional deep learning architectures (e.g., ResNet or GANs) which captures or generates the image semantic information at different layers. This difference inspires us to explore the time-wise diffusion models. We initially investigate the key contributions of the U-Net parameters to the denoising process and identify that properly zeroing out certain parameters (including large parameters) contributes to denoising, substantially improving the generation quality on the fly. Capitalizing on this discovery, we propose a simple yet effective method-termed ``MaskUNet''- that enhances generation quality with negligible parameter numbers. Our method fully leverages timestep- and sample-dependent effective U-Net parameters. To optimize MaskUNet, we offer two fine-tuning strategies: a training-based approach and a training-free approach, including tailored networks and optimization functions. In zero-shot inference on the COCO dataset, MaskUNet achieves the best FID score and further demonstrates its effectiveness in downstream task evaluations. Project page: https://gudaochangsheng.github.io/MaskUnet-Page/