Simulating Brain Signals: Creating Synthetic EEG Data via Neural-Based Generative Models for Improved SSVEP Classification

TL;DR

This paper presents neural-based generative models to create synthetic EEG data, enhancing SSVEP classification and cross-subject generalization in Brain-Computer Interface applications.

Contribution

It introduces a novel approach of using neural generative models trained on limited EEG data to produce realistic synthetic signals for improved classification.

Findings

Synthetic data improves SSVEP classification accuracy.

Cross-subject generalization increases by over 35%.

Generative models effectively augment EEG datasets.

Abstract

Despite significant recent progress in the area of Brain-Computer Interface (BCI), there are numerous shortcomings associated with collecting Electroencephalography (EEG) signals in real-world environments. These include, but are not limited to, subject and session data variance, long and arduous calibration processes and predictive generalisation issues across different subjects or sessions. This implies that many downstream applications, including Steady State Visual Evoked Potential (SSVEP) based classification systems, can suffer from a shortage of reliable data. Generating meaningful and realistic synthetic data can therefore be of significant value in circumventing this problem. We explore the use of modern neural-based generative models trained on a limited quantity of EEG data collected from different subjects to generate supplementary synthetic EEG signal vectors, subsequently utilised to train an SSVEP classifier. Extensive experimental analysis demonstrates the efficacy of our generated data, leading to improvements across a variety of evaluations, with the crucial task of cross-subject generalisation improving by over 35% with the use of such synthetic data.

Tables

Table 1. TABLE I : Classification test accuracy using generated and real-world data used to train the classifier. The baseline result is the classification of only real data.

Method	S01	S02	S03	Across Subjects
Baseline	0.91 $\pm$ 0.04	0.87 $\pm$ 0.10	0.84 $\pm$ 0.03	0.69 $\pm$ 0.03
DCGAN	0.86 $\pm$ 0.01	0.80 $\pm$ 0.04	0.89 $\pm$ 0.03	0.72 $\pm$ 0.03
WGAN	0.93 $\pm$ 0.04	0.90 $\pm$ 0.04	0.87 $\pm$ 0.02	0.71 $\pm$ 0.04
VAE	0.92 $\pm$ 0.03	0.90 $\pm$ 0.03	0.79 $\pm$ 0.03	0.67 $\pm$ 0.02

Table 2. TABLE II : Accuracy for test classification using synthetic data for pre-training stage. The baseline contains no pre-training.

Method	S01	S02	S03	Across Subjects
Baseline	0.91 $\pm$ 0.04	0.87 $\pm$ 0.10	0.84 $\pm$ 0.03	0.69 $\pm$ 0.03
DCGAN	0.97 $\pm$ 0.03	0.93 $\pm$ 0.03	0.87 $\pm$ 0.01	0.70 $\pm$ 0.02
WGAN	0.93 $\pm$ 0.03	0.90 $\pm$ 0.04	0.87 $\pm$ 0.03	0.72 $\pm$ 0.03
VAE	0.93 $\pm$ 0.03	0.92 $\pm$ 0.06	0.87 $\pm$ 0.05	0.73 $\pm$ 0.03

Table 3. TABLE III : Classification test accuracy for cross-subject Generalisation (see Figure 7(b) ) via pre-training using synthetic data. The baseline does not include a pre-training stage.

Method	S01	S02	S03	Mean
Baseline	0.35	0.54	0.42	0.45 $\pm$ 0.08
DCGAN	0.42	0.71	0.57	0.57 $\pm$ 0.12
WGAN	0.40	0.60	0.50	0.56 $\pm$ 0.13
VAE	0.70	0.90	0.85	0.82 $\pm$ 0.08

Equations

$$\underset{G}{\min }\underset{C\in \mathcal{F}}{\max }{\mathbb{E}}_{x\sim {\mathbb{P}}_r}[C(x)]-{\mathbb{E}}_{\tilde{x}\sim {\mathbb{P}}_g}[C(\tilde{x})],\text{vspace}-0.25cm$$

$$\text{centering}{\mathbb{E}}_{\hat{x}\sim {\mathbb{P}}_{\hat{x}}}[(||{▽}_{\hat{x}}C(\hat{x})|{|}_2-1{)}^2],\text{@add@centering}$$

$$L=\underset{G}{\min }\underset{D}{\max }{\mathbb{E}}_{x\sim {\mathbb{P}}_r}[log(D(x))]+{\mathbb{E}}_{\tilde{x}\sim {\mathbb{P}}_g}[log(1-D(\tilde{x}))],$$

$$\text{centering}\begin{array}{rl}L=\underset{G}{\min }\underset{C}{\max }& {\mathbb{E}}_{\tilde{x}\sim {\mathbb{P}}_g}[C(\tilde{x})]-{\mathbb{E}}_{x\sim {\mathbb{P}}_r}[C(x)]+\\ & \lambda {\mathbb{E}}_{\hat{x}\sim {\mathbb{P}}_{\hat{x}}}[(||{▽}_{\hat{x}}C(\hat{x})|{|}_2-1{)}^2],\end{array}\text{@add@centering}$$

$$L=\mathbb{E}_{q(z|x)}\Big{[}log(p(x|z))\Big{]}-KL(q(z|x)||p(z)),$$

Full text

I Introduction

Electroencephalography (EEG) is the most prominent signal acquisition approach employed for Brain-Computer Interface (BCI) as it has the non-invasive ability to capture electrical activity of the human cerebral cortex [1]. BCI is a system that translates such acquired signals to provide a communication and control medium between the human brain and external devices. BCI has received significant attention within the research community for decades [2]. However, not many BCI applications are tractable for daily use in real-world scenarios, especially for important medical applications, such as assisting patients with locked-in syndrome. This is due to the numerous shortcomings and limitations within the current state of the art that lead to the low reliability and usability of BCI [3].

Deep neural networks have recently been used to improve the classification of various aspects of EEG data [4, 5, 6]. While it is essential to have access to large quantities of data for training such methods, collecting high quality EEG data has proven difficult [7, 8]. This can be for a variety of reasons including the requirement for careful per-subject and per-session calibration. This makes EEG BCI experiments time-consuming, expensive and difficult to operate within the usually short amount of time experimental subjects can perform EEG experiments [9]. In addition to these issues, there are commonly-known limitations with EEG data in general, which can severely hinder the applicability of a system dependent on such data [7]. For instance, EEG data is known to be highly subject and session variant, which leads to a long calibration process for every individual experiment [3, 2]. This further impacts any machine learning based models which are trained upon this data, as they often demonstrate poor generalisation performance across different subjects or experiential data collecting sessions [3, 6].

In machine learning, generative models have long been used to generate entirely new and realistic data points which match the distribution of a given target dataset [10]. Recent work on neural-based models such as Generative Adversarial Networks (GAN) and Variational Auto-Encoders (VAE) have demonstrated that these are highly capable at capturing key elements from a diverse range of datasets to generate realistic samples [11]. Increasingly, there is evidence that using synthetic data, taken from a generative model, can be used as a form of data augmentation to help improve the performance of any down-stream data classification task [12].

In this paper, we detail the generation of new synthetic EEG data using a selection of customised neural-based generative models and explore applications of such data including using it to boost classification accuracy on real datasets. Uniquely, we specifically focus on dry-EEG data containing Steady State Visual Evoked Potential (SSVEP) signals. Dry-EEG requires no conductive gel which improves its usability within a BCI context, eliminating major limitations of the wet EEG systems [13, 14, 15]. However, dry-EEG results in high impedance values that cause more noise and artefacts in the data, leading to more challenging signal decoding and classification [13].

SSVEP is a type of evoked potential stimuli generated by having repeated flashes at certain frequencies presented to subjects (the flashing can occur in a video as seen in Figure 1) [1]. The frequency of the flashing is present in the EEG signals recorded from the subjects and can be extracted via a variety of competing signal processing techniques [16]. SSVEP has many important applications in BCI, for example it can be utilised to allow people with severe physical disabilities to control or communicate with external devices just by having them fixate on a flickering stimuli [17], controlling an exoskeleton [4] or navigating a humanoid robot [18]. To the best of our knowledge, this is the first work in the literature to explore the use of neural-based generative models to create dry-EEG data containing SSVEP information. In summary, we make the following major contributions in this work:

• •

The generation of synthetic dry-EEG data containing SSVEP signals using a variety of unsupervised models.

• •

A demonstration that using generated data can improve the classification of real-world EEG data, taken from multiple subjects and recorded under various conditions and sessions.

• •

An exploration of both classifier pre-training and dataset augmentation as use cases for the generated data.

• •

Further demonstration that using synthetic EEG data can increase the convergence rate of classification models, thus resulting in the observation that smaller quantities of real-world training data is required.

We perform extensive experiential evaluations to validate our claims and to aid reproducibility, we release our Python-based (PyTorch) implementation of all the generative models, along with sample input data111https://github.com/nikk-nikaznan/SSVEP-Neural-Generative-Models.

II Related Work

We consider the background information relevant to this work within two distinct areas:- recent advances made in generative models (Section II-A) and existing work on using such models to generate meaningful and coherent synthetic EEG data (Section II-B).

II-A Neural-Based Generative Models

Generative models have been proven very powerful within the context of unsupervised learning where the model learns a hidden structure of the data from its distribution to generate new data samples within the same distribution [19]. This generated dataset often contains enough variation to support the down-stream training of a secondary model [8].

Generative Adversarial Networks (GAN) [11] are capable of producing semantically sound artificial samples by inducing a competition between a generator ($G$), which attempts to capture the distribution, and a discriminator ($D$), which assesses the generator output and penalizes unrealistic samples. Both networks are trained simultaneously to achieve an equilibrium.

Training a GAN is known to be challenging with pervasive instability issues [20]. One such issue stems from the discriminator rapidly reaching optimality and effortlessly distinguishing between the fake samples output by the generator and samples from the real distribution. This will lead to a lack of meaningful gradients for training, effectively ceasing any progress towards the equilibrium.

The Wasserstein GAN (WGAN) is consequently proposed in [21] to rectify some of the issues associated with training a GAN. The Wasserstein-1 metric is used to measure the distance between the real and model distributions. Also known as the Earth Mover’s distance, ($EM(p,q)$), this metric is the minimum cost of moving distribution elements (earth mass) to transform a distribution $q$ to distribution $p$ (cost = mass $\times$ transport distance).

The Wasserstein GAN [21] has an aptly named critic ($C$) instead of a discriminator since this network is no longer a classifier. Using the EM distance, the critic will not only determine whether a sample is fake or real as a discrete binary decision, but how real or how fake the generated sample is as a continuous regressive output. Under the right training circumstances, the critic will eventually converge to a linear function with ever-present meaningful gradients and cannot saturate. The loss function in the Wasserstein GAN is created via the Kantorovich-Rubinstein duality [21]:

[TABLE]

where $\mathcal{F}$ is the set of 1-Lipschitz functions, $\mathbb{P}_{r}$ the real distribution, $\mathbb{P}_{g}$ the model distribution defined by $\tilde{x}=G(z),z\sim p(z)$, and $z$ the random noise. If $C$ is optimal, minimizing the value function with respect to $G$ minimizes $EM(\mathbb{P}_{r},\mathbb{P}_{g})$.

The Wasserstein GAN does not suffer from vanishing gradients or mode collapse. However, to guarantee continuity, a Lipschitz constraint must be enforced, which is achieved in [21] by clamping the weights. This results in the creation of a new hyper-parameter, which needs to be carefully tuned to the distribution.

A gradient norm penalty with respect to the critic input is consequently proposed in [22] to replace clamping. Since a differentiable function is 1-Lipschitz if and only if its gradient norm is no more than 1 everywhere, [22] limits the critic gradient norm by penalizing the function on the gradient norm for samples $\hat{x}\sim\mathbb{P}_{\hat{x}}$, where $\hat{x}=\epsilon x+(1-\epsilon)\tilde{x},\ 0<\epsilon<1$. This penalty term which is added to the function in Eqn. 1 is therefore as follows [22]:

[TABLE]

where $\mathbb{P}_{\hat{x}}$ is implicitly defined to sample uniformly along straight lines between pairs of points sampled from the real data distribution, $\mathbb{P}_{r}$, and $\mathbb{P}_{g}$ is the model distribution defined by $\tilde{x}=G(z),z\sim p(z)$ [22].

Auto-encoders have long been used as a method of creating a low-dimensional representation $z$ of data using an encoder model, which can be used to reconstruct the original data with minimal errors via a decoder model [23]. However, traditionally they cannot be explicitly used to generate new data samples based on the learned data distribution. Variational Auto-encoders (VAE) utilise ideas from Bayesian inference to produce a more expressive data representation, whilst also having the ability to generate new data samples [24, 25]. Unlike non-probabilistic auto-encoders [23], a VAE does not learn a fixed value for each element in $z$ but instead each element is sampled from a probability distribution before being passed to the decoder model. This has been shown to produce a more semantically meaningful representation, where individual dimensions in the hidden space can correspond to tangible elements in the dataset, such as facial expression in a dataset of human faces [26].

As the decoder model of the VAE is trained to take a sample from a Gaussian distribution and produce a realistic output, it can be used to produced new data by simply sampling points in the distribution and reconstructing them.

II-B Literature Review

Within the existing body of work in the literature, there are instances of generative models used to create synthetic EEG data. For instance, [27] proposes using the improved WGAN with GP for a single channel of the EEG data for motor imagery task. The generator consists of one linear layer, six convolutional layers where each layer consists of an up-sampling operation, two convolutions and one fully-connected layer. The authors evaluate the performance of four models, three different up-sampling methods (nearest-neighbour, linear interpolation and cubic interpolation with convolutional down-sampling) and down-sampling via average pooling with the original WGAN-GP. All the models are then assessed using four evaluation metrics; inception score, Frechet inception distance, Euclidean distance and sliced Wasserstein distance. All four models are demonstrated to outperform WGAN-GP.

In [9], the authors propose deep EEG super-resolution using a GAN. The model is applied to a small number of EEG channel data to interpolate other channel signals using motor imagery dataset from [28]. The super-resolution data (SR) is generated via WGAN with convolutional filters using the low-resolution data (LR) from the dataset by down-sampling the EEG channels by scale factors of two and four for two different experiments. The channels removed from the down-sampling processes are used as high-resolution data (HR) to compete against SR in the discriminator. They evaluate the performance of the SR by performing classification and comparing the accuracy with the classification performed using HR. The authors conclude that SR is capable of producing high spatial resolution EEG signals from low resolution signals.

Instead of generating EEG signals, [29] generates previously seen images while having brain signals recorded by EEG. Six subjects are shown 2000 images with 40 classes per subject. The EEG signals are pre-processed by hardware notch filter between 49 and 51 Hz and bandpass filter between 14 and 70 Hz. Using LSTM RNN, the temporal feature representations are encoded, which are subsequently used as condition vectors employed along with random noise vectors to generate new images. They obtain a test accuracy of 83.9% when evaluating the LSTM model for feature representation.

In [7], synthetic EEG signals are generated by inverting the artificial time-frequency representation (TFR) obtained from conditional Deep Convolutional GAN (cDCGAN). The Wavelet Transform is used to obtain the TFR of the signals to be used by the cDCGAN to generate the artificial TFR of the EEG signals. The BCI competition II dataset III [30] is used as the EEG data. They evaluate the efficacy of the synthetic data by comparing the classification accuracy of the model trained on real data, synthetic data and a mixture of real and synthetic data with different ratios. Using additional synthetic data, the accuracy improved up to 3$\%$.

In [8] and [31], a GAN is used to generate synthetic emotion recognition EEG data. The authors in [8] generate EEG in the form of differential entropy from noise distribution using a conditional WGAN with two emotion recognition datasets. They evaluate the performance of the generated data by combining the synthetic and real-world data to train a classification model compared against a model solely trained on real-world data. The addition of the synthetic data leads to an improvement in the accuracy of up to 20$\%$ for different datasets.

The approach in [31] combines classification and generative networks in a model using an Auxiliary Classifier GAN (AC-GAN). Part of the approach is encoding the data from SEED and DEAP datasets into images using a Markov Transition Field (MTF) before passing them into the AC-GAN to generate new data samples. Every sample comes with a corresponding class label. A Tiled CNN is employed to classify the MTF images to either fake or real and with the class label. They improved the classification by less than 1$\%$ in the SEED dataset as compared with previous work.

In this paper, we generate synthetic EEG signals from SSVEP tasks using dry-EEG. While earlier work [7, 31] has been able to generate features within a secondary domain, from which EEG signals are subsequently reconstructed, we are the first to introduce the concept of generating meaningful EEG data directly in signal space via end-to-end training instead of first transforming the signals into different domains for SSVEP classification. This improves overall efficiency, removes any need for additional signal processing, and prevents potential instability errors introduced during transformations.

III Proposed Approach

As one of the objectives of this work is to create synthetic EEG data, which can be used to improve the training of a downstream classifier [5], we explore the use of a variety of neural-based generative models (as described in Section II). In this section, a brief description of the details of the models used in this work is provided.

III-A Deep Convolutional Generative Adversarial Network

As part of this work, we use the generative model proposed in [32]. Random noise vectors $z$ are sampled from a Gaussian distribution and used as the generator input. At every iteration, the generator outputs fake data samples ($\tilde{x}=G(z)$), which are then passed to the discriminator along with randomly selected real data samples $x$, classifying them as either fake or real, the gradients from which are employed to train the generator, leading to higher quality outputs at every step. An overview of our generative adversarial model is seen in Figure 2.

III-A1 Loss Function

The loss function is based on the competition between the generator and the discriminator following the minimax objective [11]:

[TABLE]

where $\mathbb{P}_{r}$ is the data distribution, $\mathbb{P}_{g}$ the model distribution defined by $\tilde{x}=G(z),z\sim p(z)$, and $z$ the random noise vector used as the input to the generator. An overview of the training pipeline is seen in Figure 2.

III-A2 Implementation Details

The network architecture is based on that of [32], with the exception of the use of one dimensional convolutions since the networks process EEG signal vectors rather than images. Our generator consists of one dense layer and three 1D transpose convolutional layers.

This vector is then used as the input to our light-weight discriminator, consisting of two layers; the first containing a convolution-BatchNorm-LeakyReLU($slope=0.2$) module followed by max pooling, and the second using a fully-connected layer followed by a leaky ReLU ($slope=0.2$).

Implementation and training is carried out in PyTorch [33], with Adam [34] used as the optimization approach (momentum $\beta_{1}=0.5$, $\beta_{2}=0.999$, initial learning rate $\alpha=0.0001$).

III-B Improved Wasserstein Generative Adversarial Network

Similarly to the training procedure in Section III-A, the Wasserstein GAN is made up of two completing networks, a generator and a critic. The generator receives a random noise vector $z$ as its input, and the critic determines how real or fake the data samples created by the generator are by calculating the distance between the real data distribution and the model distribution (Earth Mover’s distance).

Since it is significantly important to keep the critic optimal at all times, we train the critic 25 times per each generator training iteration for the first 100 generator iterations and 5 times per each generator iteration for the rest of the training process.

III-B1 Loss Function

Here, we take advantage of the improved Wasserstein GAN [22] with the following loss function:

[TABLE]

where $\mathbb{P}_{g}$ is the model distribution defined by $\tilde{x}=G(z),z\sim p(z)$, $z$ is random noise, $\mathbb{P}_{r}$ is the true data distribution, and $\mathbb{P}_{\hat{x}}$ is implicitly defined to sample uniformly along straight lines between pairs of points sampled from $\mathbb{P}_{r}$ and $\mathbb{P}_{g}$.

III-B2 Implementation Details

For the sake of consistency, the architecture of the networks used here are similar to that of the networks in Section III-A. Similarly, all implementation and training is carried out in PyTorch [33], with Adam [34] used as the optimization approach (momentum $\beta_{1}=0.1$, $\beta_{2}=0.999$, initial learning rate $\alpha=0.0001$).

III-C Variational Autoencoder

In addition to the GAN based approaches, we also explore the creation of a Convolutional Variational Auto-encoder (detailed in Section II-A) to generate synthetic EEG data. Our VAE model uses 1D convolutions to encode features from a given EEG data sample, which are used to parametrise a Gaussian distribution from which a latent and compressed representation of the input data is sampled. This latent representation is passed to the decoder section of the model which comprises transposed convolutions used to transform the latent representation back into the original EEG data.

Once we have trained the VAE model using the objective function detailed in the next section, we are able to use it to generate entirely new data samples, which could have plausibly come from the training data, but which is not conditional on any input to the model.

III-C1 Loss Function

The encoder section of our VAE model is trained to learn two output vectors, $\mu$ and $\sigma$, which represent the mean and variance of the latent space from which $z$ will be sampled, $z=\mathcal{N}(\mu,\sigma)$. Using the sampled representation $z$, the decoder section of the VAE is trained to reconstruct the input data $x$. Consequently, our VAE is trained to infer the intractable distribution $p(z|x)$, that being the likelihood of $z$ given the data $\hat{x}$, using a stand-in tractable one $q(z|x)$ in the following manner:

[TABLE]

where $KL$ is the Kullback-Leibler distance between $p$ and $q$, $q(z|x)$ is the output of the convolutional based encoder portion of our VAE and $p(x|z)$ is the output from the decoder section. We make use of a Gaussian prior as the distribution for $p(z)$. Figure 8(c) shows the the general layout of our VAE model.

III-C2 Implementation Details

We utilise a convolutional encoder for our VAE model and a transpose convolution based decoder. The encoder consists of a 1D convolution, with BatchNorm and leaky ReLU ($slope=0.2$) as the activation function, followed by max pooling. This common learned feature representation is passed into two separate linear layers which learn the $\mu$ and $\sigma$ used to parametrise the Gaussian and generate $z$. The decoder architecture comprises three stacked 1D transpose convolutional layers all using leaky ReLU ($slope=0.2$) to perform the reconstruction from $z$ to create $\hat{x}$.

As with the GAN models, all implementation and training is carried out in PyTorch [33], with Adam [34] used as the optimization algorithm (momentum $\beta_{1}=0.1$, $\beta_{2}=0.999$, initial learning rate $\alpha=0.0001$).

IV Experimental Setup

This section will detail the setup of our experimental evaluation, including introducing the empirical datasets used, detailing how we generate new data samples from our generative models and our procedure for evaluating the quality of the generated data.

IV-A Empirical SSVEP Dry-EEG Datasets

We make use of two empirical SSVEP dry-EEG datasets which we collected and fully detailed in our previous work [18]. The collection procedure for this dataset is detailed in Figure 4. This empirical data is used as a way of validating that the generated data is realistic enough to be used to improve the performance of a classification model. In this dataset, objects are detected in a video sequence using a pre-trained object detection model [35]. The detected objects are then flickered by rendering black/white polygon boxes on top of the objects with display frequency modulations of 10, 12 and 15 Hz to create the frequencies, common to both datasets, which we attempt to detect via the SSVEP paradigm. Our two empirical datasets are detailed below:

• •

Video-Stimuli Dataset: This dataset is collected from an offline experiment (shown in Figure 1) and is used as a basis for our generative models to learn the distribution of the SSVEP data. This dataset comprises 50 unique samples recorded for each of the three classes, taken from one subject (S01) performing the SSVEP task by looking at objects detected in a pre-recorded video sequence.

• •

NAO Dataset: This dataset comprises data from an experiment containing both offline and online elements (highlighted in Figure 4). The offline portion of the data contains 50 unique samples for each of the three classes taken from three subjects (S01, S02, S03) performing the SSVEP task by looking at objects detected in a pre-recorded video sequence from a humanoid robot. The online portion of the data contains 30 samples per class taken from the same three subjects when navigating the robot in real time [18].

IV-B Synthetic Data Generation Procedure

Each of the generative models are trained upon data taken only from the Video-Stimuli dataset, detailed in Section IV-A. This will allow us to explore if data generated under one condition, can be used to improve the classification of data recorded under another. Once training is complete, the models are used to generate entirely new and synthetic datasets, with one dataset being made for each model. A potentially unlimited amount of data could theoretically be generated from our models. For our experiments, we generate 500 unique samples, each three seconds in length, for each of the three SSVEP frequencies present in the original Video-Stimuli dataset.

IV-C Classification Procedure

In our experimental evaluation, we investigate exploiting the generated data to improve the classification accuracy on the NAO dataset. To perform this classification, we employ a model based on the SSVEP Convolutional Unit (SCU) Convolutional Neural Network (CNN) architecture [5] (Figure 5). Prior work has shown this model to outperform traditional approaches and even time-series specific models like Recurrent Neural Networks (RNN) when classifying SSVEP EEG data [5]. The classification model used for all experiments comprises of 1D convolutional layers, with batch normalization and max pooling. We first pre-process the EEG channels by referencing the data, applying a bandpass filter between 9 to 60 Hz, a notch filter at 50 Hz and finally normalizing the data between 0 and 1.

V Results

Our experimental evaluation is designed to firstly assess if it is indeed possible to generate realistic EEG data containing SSVEP signals and secondly, if this synthetic data can be used in a variety of ways to improve the classification accuracy on other real-world empirical datasets. All the classification results presented are the mean results from five different random seeds in order to test robustness and each experiment uses identical train/test splits for all runs to allow for direct comparisons to be made.

V-A Data Visualization via Fast Fourier Transforms

SSVEP data has the phenomenon of frequency tagging, where the primary visual areas in human cortical oscillates to match the frequency of the fluctuating sinusoidal cycle of the SSVEP stimuli presented to the subject [36]. EEG signals recorded during an SSVEP task will contain the target frequency clearly identifiable in the frequency domain [16].

To validate that our generative models are indeed producing viable SSVEP signals, we visualise both the real and synthetic data via the Fast Fourier Transform (FFT) to decompose the EEG signals into the frequency domain. Figure 6 displays the frequency plot of the new synthetic data generated from our DCGAN, WGAN and VAE compared against the empirical data. As can be seen, all models accurately capture the characteristic SSVEP peaks at the target frequency and associated harmonics [1], with the VAE producing signals with a comparatively lower amplitude. This result is very encouraging as it strongly suggests generative models can produce realistic SSVEP dry-EEG data.

V-B Mixed Real and Synthetic Data Classification

To evaluate the applicability of synthetic EEG data being used to improve classification results, we combine the synthetic (using 30 samples per class) and real data into a single training set from which the classifier (see Section IV-C for details) is trained. Table I displays the results, with values given per and across all subjects, where the baseline is no synthetic data included in the training set. It is also interesting to note that no single generative models data source performs the best across all subjects.

Having a single classification model perform well across subjects is known to be highly challenging [17], yet possible [5]. Table I illustrates how the inclusion of synthetic data within the training set can positively influence generalisation capabilities across subjects, which is an important result and of significant value in real-world applications. It is also interesting to note that although the synthetic data was generated from a different stimuli and sessions, its inclusion is still capable of improving the classification accuracy.

V-C Classification with Pre-Training

To further explore the usefulness of the synthetic data, we pre-train our classifier using only synthetic data (with 500 samples generated per class) and then further fine-tune the model using the real data as in Figure 7(a). The performance on the testing set is presented in Table II, where the baseline is no pre-training using synthetic data. Figure 9 highlights how the loss values for this task vary over training epochs. It can be seen that models pre-trained using the synthetic data converge faster and to a lower overall loss value.

Both Table II and Figure 9 demonstrate the ability to use the synthetic data to pre-train the network and improve the test accuracy on real data - another key result. This demonstrates the possibility of achieving higher classification accuracy using a smaller training set of real-world data, which resolves one of the most important challenges associated with SSVEP signal classification, unavailability of large datasets.

Additional experiments were also conducted using varying quantities of synthetic data during pre-training. As seen in Figure 8, the models perform better on average when 500 synthetic data samples are used in the pre-training stage. This is primarily due to the subject and session variant nature of dry-EEG data, since the models can easily over-fit to the distribution of the synthetic data in the presence of excessive pre-training. Furthermore, Figure 8(d) demonstrates how the training time can rapidly increase when the size of the synthetic pre-training dataset grows too large. Accordingly, the use of 500 synthetic data points empirically offers an optimal trade-off between improved performance and tractable training time in this instance.

V-D Cross-Subject Generalisation

As mentioned in I, SSVEP classification models often exhibit poor generalisation performance across different subjects or experiential data collecting sessions [3, 6]. In previous work [5], we successfully classified an unseen subject with no additional training required on that subject. In this section, we also attempt to demonstrate how pre-training on synthetic data can enhance the cross-subject generalisation capabilities of the model, i.e., the ability to classify data captured from one subject using a model trained on data captured from another. We train the generative models on S0X NAO Offline data (from the Nao Dataset), pre-train the classifier using 500 samples of synthetic data output by said generative models, train the classifier on data from a different subject (S0Y) and finally test on online data from S0X from the Nao Dataset (Figure 7(b)). As commonly seen within the literature [2, 17], one of the most important challenges in EEG-based research is the properties of EEG signals that vary from one subject to another as signal features can be specific to individual subjects.

Table III demonstrates how the baseline model performs poorly when trained using data collected from S0Y but tested on data from S0X, pointing to the lack of model Generalisability when EEG data is used. However, by pre-training the classifier on the generated data using NAO offline data from S0X, we see a large improvement in classification accuracy - especially when data generated by the VAE is used. This is a significant observation in the BCI research domain leading to a conjecture that, with further improvement in the results, we can eliminate the requirement for per-subject, per-session calibration for online applications.

VI Conclusion

In this work, we exploit recent advances made in neural-based generative models to explore potential benefits they can offer within the context of SSVEP classification models trained on EEG data. Since data acquisition within real-world scenarios suffers from a variety of challenges, using synthetically generated EEG signals can prove highly beneficial in improving the accuracy, convergence rate and generalization capabilities of any model trained to classify EEG data. We generate synthetic EEG signals using three state-of-the-art generative models - a Generative Adversarial Network, a Wasserstein Generative Adversarial Network and a Variational Auto-Encoder. Extensive evaluations demonstrate the efficacy of the synthetic data generated by said models across multiple experimental setups, with the inclusion of the generated data always improving the results.

Future work will investigate the the influence of the quantity of the synthetic data generated using different approaches used during pre-training on the classification results can reveal valuable insight into the inner-workings of the classification process and nature of the synthetic data. Furthermore, we also plan to assess if mixing generated data taken from several models for pre-training can also improve our results.