On Learning Representations for Tabular Data Distillation

TL;DR

This paper introduces TDColER, a novel framework for tabular data distillation using column embeddings, and presents TDBench, a benchmark for evaluating such methods, demonstrating significant improvements in data quality across various models.

Contribution

The paper proposes TDColER, a new approach for tabular data distillation that addresses feature heterogeneity and non-differentiable models, and introduces TDBench, a comprehensive benchmark for evaluation.

Findings

TDColER improves data quality by up to 143% across models.

Created TDBench with 226,890 datasets and 548,880 models.

Significant enhancement over existing distillation schemes.

Abstract

Dataset distillation generates a small set of information-rich instances from a large dataset, resulting in reduced storage requirements, privacy or copyright risks, and computational costs for downstream modeling, though much of the research has focused on the image data modality. We study tabular data distillation, which brings in novel challenges such as the inherent feature heterogeneity and the common use of non-differentiable learning models (such as decision tree ensembles and nearest-neighbor predictors). To mitigate these challenges, we present $\texttt{TDColER}$, a tabular data distillation framework via column embeddings-based representation learning. To evaluate this framework, we also present a tabular data distillation benchmark, ${{\sf \small TDBench}}$. Based on an elaborate evaluation on ${{\sf \small TDBench}}$, resulting in 226,890 distilled datasets and 548,880 models trained on them, we demonstrate that $\texttt{TDColER}$ is able to boost the distilled data quality of off-the-shelf distillation schemes by 0.5-143% across 7 different tabular learning models.