Bag of Design Choices for Inference of High-Resolution Masked Generative Transformer

TL;DR

This paper investigates and optimizes inference techniques for masked generative Transformers in high-resolution image generation, providing practical design choices that improve performance and sampling efficiency.

Contribution

It introduces and analyzes specific inference strategies for MGT, filling a gap in existing research, and demonstrates their effectiveness through extensive experiments.

Findings

Enhanced inference techniques improve sampling quality.

Design choices lead to ~70% winning rate on HPS v2.

Sampling acceleration methods increase efficiency.

Abstract

Text-to-image diffusion models (DMs) develop at an unprecedented pace, supported by thorough theoretical exploration and empirical analysis. Unfortunately, the discrepancy between DMs and autoregressive models (ARMs) complicates the path toward achieving the goal of unified vision and language generation. Recently, the masked generative Transformer (MGT) serves as a promising intermediary between DM and ARM by predicting randomly masked image tokens (i.e., masked image modeling), combining the efficiency of DM with the discrete token nature of ARM. However, we find that the comprehensive analyses regarding the inference for MGT are virtually non-existent, and thus we aim to present positive design choices to fill this gap. We propose and redesign a set of enhanced inference techniques tailored for MGT, providing a detailed analysis of their performance. Additionally, we explore several DM-based approaches aimed at accelerating the sampling process on MGT. Extensive experiments and empirical analyses on the recent SOTA MGT, such as MaskGIT and Meissonic lead to concrete and effective design choices, and these design choices can be merged to achieve further performance gains. For instance, in terms of enhanced inference, we achieve winning rates of approximately 70% compared to vanilla sampling on HPS v2 with Meissonic-1024x1024.