TreeDiff: AST-Guided Code Generation with Diffusion LLMs

TL;DR

TreeDiff introduces a syntax-aware diffusion approach for code generation, leveraging AST structures to improve the model's understanding of syntax and dependencies, resulting in significant performance gains.

Contribution

The paper presents a novel AST-guided diffusion framework that incorporates structural priors into the corruption process for improved code generation.

Findings

Achieves 13.3% relative improvement over random masking methods.

Effectively captures syntactic boundaries and long-range dependencies.

Enhances the internalization of programming language structure.

Abstract

Code generation is increasingly critical for real-world applications. Still, diffusion-based large language models continue to struggle with this demand. Unlike free-form text, code requires syntactic precision; even minor structural inconsistencies can render a program non-executable. Existing diffusion-based large language models rely on random token masking for corruption, leading to two key failures: they lack awareness of syntactic boundaries during the iterative denoising process, and they fail to capture the long-range hierarchical dependencies essential for program correctness. We propose TreeDiff to address both issues. Specifically, we propose a syntax-aware diffusion framework that incorporates structural priors from Abstract Syntax Tree (AST) into the corruption process. Instead of masking individual tokens at random, we selectively mask tokens belonging to key AST nodes. By aligning the corruption process with the underlying structure of code, our method encourages the model to internalize the compositional nature of programming languages, enabling it to reconstruct programs that respect grammatical boundaries and capture long-range dependencies. Our method achieves a 13.3% relative improvement over the random masking training method, demonstrating its effectiveness in code generation task by leveraging underlying structures.