New Properties of the Data Distillation Method When Working With Tabular Data

TL;DR

This paper explores a data distillation algorithm adapted for tabular data, demonstrating that models trained on distilled samples can outperform those trained on original data, and proposes using multiple architectures to improve generalization.

Contribution

It extends a data distillation method from image to tabular data and introduces multi-architecture training to enhance generalization across models.

Findings

Distilled tabular data can lead to better model performance.

Using multiple architectures improves generalization of distilled data.

The method shows promise for reducing training data volume without sacrificing accuracy.

Abstract

Data distillation is the problem of reducing the volume oftraining data while keeping only the necessary information. With thispaper, we deeper explore the new data distillation algorithm, previouslydesigned for image data. Our experiments with tabular data show thatthe model trained on distilled samples can outperform the model trainedon the original dataset. One of the problems of the considered algorithmis that produced data has poor generalization on models with differenthyperparameters. We show that using multiple architectures during distillation can help overcome this problem.