Designing Pre-training Datasets from Unlabeled Data for EEG Classification with Transformers

TL;DR

This paper introduces a method to create labeled EEG datasets from unlabeled data to pre-train transformer models, significantly improving training efficiency and accuracy in epileptic seizure forecasting.

Contribution

The paper presents a novel approach to generate labeled datasets from unlabeled EEG data for pre-training transformers, enhancing model performance and training speed.

Findings

Pre-trained models trained with the proposed method train over 50% faster.

Pre-trained models achieve higher accuracy (92.16% vs. 90.93%).

Pre-training improves AUC from 0.9648 to 0.9702.

Abstract

Transformer neural networks require a large amount of labeled data to train effectively. Such data is often scarce in electroencephalography, as annotations made by medical experts are costly. This is why self-supervised training, using unlabeled data, has to be performed beforehand. In this paper, we present a way to design several labeled datasets from unlabeled electroencephalogram (EEG) data. These can then be used to pre-train transformers to learn representations of EEG signals. We tested this method on an epileptic seizure forecasting task on the Temple University Seizure Detection Corpus using a Multi-channel Vision Transformer. Our results suggest that 1) Models pre-trained using our approach demonstrate significantly faster training times, reducing fine-tuning duration by more than 50% for the specific task, and 2) Pre-trained models exhibit improved accuracy, with an increase from 90.93% to 92.16%, as well as a higher AUC, rising from 0.9648 to 0.9702 when compared to non-pre-trained models.