Exploring Transformer Placement in Variational Autoencoders for Tabular Data Generation

TL;DR

This paper investigates how integrating Transformers into Variational Autoencoders affects tabular data generation, revealing trade-offs in data fidelity and diversity, and noting high similarity between Transformer blocks.

Contribution

It provides an empirical analysis of Transformer placement within VAEs for tabular data, highlighting their impact on model performance and internal representations.

Findings

Transformers improve modeling of complex feature interactions.

Positioning Transformers affects the balance between fidelity and diversity.

Transformer blocks exhibit high similarity and near-linear relationships in the decoder.

Abstract

Tabular data remains a challenging domain for generative models. In particular, the standard Variational Autoencoder (VAE) architecture, typically composed of multilayer perceptrons, struggles to model relationships between features, especially when handling mixed data types. In contrast, Transformers, through their attention mechanism, are better suited for capturing complex feature interactions. In this paper, we empirically investigate the impact of integrating Transformers into different components of a VAE. We conduct experiments on 57 datasets from the OpenML CC18 suite and draw two main conclusions. First, results indicate that positioning Transformers to leverage latent and decoder representations leads to a trade-off between fidelity and diversity. Second, we observe a high similarity between consecutive blocks of a Transformer in all components. In particular, in the decoder, the relationship between the input and output of a Transformer is approximately linear.