Unlocking the Capabilities of Masked Generative Models for Image Synthesis via Self-Guidance

TL;DR

This paper introduces a self-guidance sampling method for masked generative models (MGMs) that improves image quality and diversity, outperforming existing methods with efficient training and sampling.

Contribution

It extends guidance methods to MGMs and proposes a self-guidance approach using semantic smoothing, enhancing image synthesis performance.

Findings

Self-guidance improves image quality and diversity in MGMs.

Proposed method outperforms existing sampling techniques.

Achieves better quality-diversity trade-off with efficient training.

Abstract

Masked generative models (MGMs) have shown impressive generative ability while providing an order of magnitude efficient sampling steps compared to continuous diffusion models. However, MGMs still underperform in image synthesis compared to recent well-developed continuous diffusion models with similar size in terms of quality and diversity of generated samples. A key factor in the performance of continuous diffusion models stems from the guidance methods, which enhance the sample quality at the expense of diversity. In this paper, we extend these guidance methods to generalized guidance formulation for MGMs and propose a self-guidance sampling method, which leads to better generation quality. The proposed approach leverages an auxiliary task for semantic smoothing in vector-quantized token space, analogous to the Gaussian blur in continuous pixel space. Equipped with the parameter-efficient fine-tuning method and high-temperature sampling, MGMs with the proposed self-guidance achieve a superior quality-diversity trade-off, outperforming existing sampling methods in MGMs with more efficient training and sampling costs. Extensive experiments with the various sampling hyperparameters confirm the effectiveness of the proposed self-guidance.