Partition Generative Modeling: Masked Modeling Without Masks

TL;DR

Partition Generative Models (PGMs) replace masking with partitioning to enable parallel, any-order token generation while processing only relevant tokens during sampling, achieving higher throughput and comparable quality to existing models.

Contribution

PGMs introduce a novel partitioning approach that eliminates masks, combining the advantages of MGMs and ARMs for efficient, flexible generative modeling.

Findings

PGMs achieve 5-5.5x higher throughput on OpenWebText.

PGMs produce lower perplexity samples than MDLM.

On ImageNet, PGMs reach comparable FID with 7.5x throughput improvement.

Abstract

Masked generative models (MGMs) can generate tokens in parallel and in any order, unlike autoregressive models (ARMs), which decode one token at a time, left-to-right. However, MGMs process the full-length sequence at every sampling step, including mask tokens that carry no information. In contrast, ARMs process only the previously generated tokens. We introduce ``Partition Generative Models'' (PGMs), which replace masking with partitioning. Tokens are split into two groups that cannot attend to each other, and the model learns to predict each group conditioned on the other, eliminating mask tokens entirely. Because the groups do not interact, PGMs can process only the clean tokens during sampling, like ARMs, while retaining parallel, any-order generation, like MGMs. On OpenWebText, PGMs achieve $5-5.5\times$ higher throughput than MDLM while producing samples with lower Generative Perplexity. On ImageNet, PGMs reach comparable FID to MaskGIT with a $7.5\times$ throughput improvement. With twice as many steps, the FID improves to 4.56 while remaining $3.9\times$ faster than MGMs. Finally, PGMs remain compatible with existing MGM samplers and distillation methods.