# Scaffold Diffusion: Sparse Multi-Category Voxel Structure Generation with Discrete Diffusion

## TL;DR

This paper introduces Scaffold Diffusion, a novel discrete diffusion model for generating realistic, sparse, multi-category 3D voxel structures, effectively handling class imbalance and high sparsity in 3D data.

## Contribution

It extends discrete diffusion language models to 3D spatial data, enabling coherent structure generation beyond sequential domains.

## Key findings

- Produces realistic 3D structures with over 98% sparsity
- Outperforms prior baselines and auto-regressive models
- Demonstrates spatial coherence in generated voxel structures

## Abstract

Generating realistic sparse multi-category 3D voxel structures is difficult due to the cubic memory scaling of voxel structures and moreover the significant class imbalance caused by sparsity. We introduce Scaffold Diffusion, a generative model designed for sparse multi-category 3D voxel structures. By treating voxels as tokens, Scaffold Diffusion uses a discrete diffusion language model to generate 3D voxel structures. We show that discrete diffusion language models can be extended beyond inherently sequential domains such as text to generate spatially coherent 3D structures. We evaluate on Minecraft house structures from the 3D-Craft dataset and demonstrate that, unlike prior baselines and an auto-regressive formulation, Scaffold Diffusion produces realistic and coherent structures even when trained on data with over 98% sparsity. We provide an interactive viewer where readers can visualize generated samples and the generation process: https://scaffold.deepexploration.org/