Discrete Flow Matching

TL;DR

Discrete Flow Matching introduces a new discrete generative paradigm that effectively models high-dimensional discrete data like language, improving sample quality and scaling to large models.

Contribution

It proposes a novel discrete flow framework with flexible probability paths, a unified sampling formula, and demonstrates improved performance on coding benchmarks.

Findings

Achieves 6.7% Pass@1 on HumanEval

Reaches 13.4% Pass@10 on HumanEval

Scales up to 1.7B parameters

Abstract

Despite Flow Matching and diffusion models having emerged as powerful generative paradigms for continuous variables such as images and videos, their application to high-dimensional discrete data, such as language, is still limited. In this work, we present Discrete Flow Matching, a novel discrete flow paradigm designed specifically for generating discrete data. Discrete Flow Matching offers several key contributions:(i) it works with a general family of probability paths interpolating between source and target distributions; (ii) it allows for a generic formula for sampling from these probability paths using learned posteriors such as the probability denoiser ($x$-prediction) and noise-prediction ($\epsilon$-prediction); (iii) practically, focusing on specific probability paths defined with different schedulers improves generative perplexity compared to previous discrete diffusion and flow models; and (iv) by scaling Discrete Flow Matching models up to 1.7B parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1 and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of generating high-quality discrete data in a non-autoregressive fashion, significantly closing the gap between autoregressive models and discrete flow models.