Diffusion-nested Auto-Regressive Synthesis of Heterogeneous Tabular Data

TL;DR

This paper introduces TabDAR, a novel model combining diffusion and autoregressive techniques with masked transformers to generate heterogeneous tabular data flexibly, handling continuous and discrete features and arbitrary column orders.

Contribution

The paper proposes a new diffusion-nested autoregressive model for tabular data that addresses heterogeneity and permutation invariance, enabling flexible and high-quality data synthesis.

Findings

Outperforms previous methods by 18-45% on multiple metrics

Handles both continuous and discrete data effectively

Supports unconditional and conditional sampling

Abstract

Autoregressive models are predominant in natural language generation, while their application in tabular data remains underexplored. We posit that this can be attributed to two factors: 1) tabular data contains heterogeneous data type, while the autoregressive model is primarily designed to model discrete-valued data; 2) tabular data is column permutation-invariant, requiring a generation model to generate columns in arbitrary order. This paper proposes a Diffusion-nested Autoregressive model (TabDAR) to address these issues. To enable autoregressive methods for continuous columns, TabDAR employs a diffusion model to parameterize the conditional distribution of continuous features. To ensure arbitrary generation order, TabDAR resorts to masked transformers with bi-directional attention, which simulate various permutations of column order, hence enabling it to learn the conditional distribution of a target column given an arbitrary combination of other columns. These designs enable TabDAR to not only freely handle heterogeneous tabular data but also support convenient and flexible unconditional/conditional sampling. We conduct extensive experiments on ten datasets with distinct properties, and the proposed TabDAR outperforms previous state-of-the-art methods by 18% to 45% on eight metrics across three distinct aspects.