Tabular data generation with tensor contraction layers and transformers

TL;DR

This paper explores the use of tensor contraction layers and transformers within Variational Autoencoders to improve tabular data generation, addressing the challenges of mixed feature types and intra-variable relationships.

Contribution

It introduces and empirically evaluates hybrid VAE architectures incorporating tensor contraction layers and transformers for better tabular data modeling.

Findings

Tensor contraction layers enhance density estimation.

Hybrid models perform competitively in machine learning tasks.

Embedding representations improve modeling of tabular data.

Abstract

Generative modeling for tabular data has recently gained significant attention in the Deep Learning domain. Its objective is to estimate the underlying distribution of the data. However, estimating the underlying distribution of tabular data has its unique challenges. Specifically, this data modality is composed of mixed types of features, making it a non-trivial task for a model to learn intra-relationships between them. One approach to address mixture is to embed each feature into a continuous matrix via tokenization, while a solution to capture intra-relationships between variables is via the transformer architecture. In this work, we empirically investigate the potential of using embedding representations on tabular data generation, utilizing tensor contraction layers and transformers to model the underlying distribution of tabular data within Variational Autoencoders. Specifically, we compare four architectural approaches: a baseline VAE model, two variants that focus on tensor contraction layers and transformers respectively, and a hybrid model that integrates both techniques. Our empirical study, conducted across multiple datasets from the OpenML CC18 suite, compares models over density estimation and Machine Learning efficiency metrics. The main takeaway from our results is that leveraging embedding representations with the help of tensor contraction layers improves density estimation metrics, albeit maintaining competitive performance in terms of machine learning efficiency.