Technical Report on Text Dataset Distillation

TL;DR

This report reviews the development of text dataset distillation, highlighting key methods, challenges, and recent advances in creating smaller, effective synthetic text datasets for training models.

Contribution

It provides a comprehensive overview of past and recent methods in text dataset distillation, emphasizing challenges and future directions.

Findings

Progress in transformer-based and discrete text generation methods

Identification of key challenges like benchmarking and discrete data handling

Recognition of the field's ongoing maturation and potential for improvement

Abstract

In the vision domain, dataset distillation arises as a technique to condense a large dataset into a smaller synthetic one that exhibits a similar result in the training process. While image data presents an extensive literature of distillation methods, text dataset distillation has fewer works in comparison. Text dataset distillation initially grew as an adaptation of efforts from the vision universe, as the particularities of the modality became clear obstacles, it rose into a separate branch of research. Several milestones mark the development of this area, such as the introduction of methods that use transformer models, the generation of discrete synthetic text, and the scaling to decoder-only models with over 1B parameters. Despite major advances in modern approaches, the field remains in a maturing phase, with room for improvement on benchmarking standardization, approaches to overcome the discrete nature of text, handling complex tasks, and providing explicit examples of real-world applications. In this report, we review past and recent advances in dataset distillation for text, highlighting different distillation strategies, key contributions, and general challenges.