EEG-DLite: Dataset Distillation for Efficient Large EEG Model Training

TL;DR

EEG-DLite introduces a data distillation method that efficiently reduces EEG datasets by removing noise and redundancy, enabling effective large EEG model pre-training with significantly less data.

Contribution

This work presents the first systematic EEG data distillation framework that improves training efficiency without sacrificing model performance.

Findings

Training on 5% of the distilled dataset matches full dataset performance.

EEG-DLite reduces training data volume while maintaining diversity.

Distilled datasets lead to comparable or better downstream task results.

Abstract

Large-scale EEG foundation models have shown strong generalization across a range of downstream tasks, but their training remains resource-intensive due to the volume and variable quality of EEG data. In this work, we introduce EEG-DLite, a data distillation framework that enables more efficient pre-training by selectively removing noisy and redundant samples from large EEG datasets. EEG-DLite begins by encoding EEG segments into compact latent representations using a self-supervised autoencoder, allowing sample selection to be performed efficiently and with reduced sensitivity to noise. Based on these representations, EEG-DLite filters out outliers and minimizes redundancy, resulting in a smaller yet informative subset that retains the diversity essential for effective foundation model training. Through extensive experiments, we demonstrate that training on only 5 percent of a 2,500-hour dataset curated with EEG-DLite yields performance comparable to, and in some cases better than, training on the full dataset across multiple downstream tasks. To our knowledge, this is the first systematic study of pre-training data distillation in the context of EEG foundation models. EEG-DLite provides a scalable and practical path toward more effective and efficient physiological foundation modeling. The code is available at https://github.com/t170815518/EEG-DLite.