Uncertainty Quantification for Motor Imagery BCI -- Machine Learning vs. Deep Learning

TL;DR

This paper compares uncertainty quantification methods in motor imagery BCI classifiers, showing classical methods like CSP-LDA and MDRM with temperature scaling outperform deep learning models in confidence estimation, while deep learning models excel in classification accuracy.

Contribution

It provides a comprehensive comparison of classical and deep learning classifiers for BCI, highlighting the strengths of traditional methods in uncertainty estimation and deep learning in classification performance.

Findings

CSP-LDA and MDRM provide well-calibrated uncertainty estimates.

Deep learning models achieve higher classification accuracy.

All models can distinguish between easy and difficult samples.

Abstract

Brain-computer interfaces (BCIs) turn brain signals into functionally useful output, but they are not always accurate. A good Machine Learning classifier should be able to indicate how confident it is about a given classification, by giving a probability for its classification. Standard classifiers for Motor Imagery BCIs do give such probabilities, but research on uncertainty quantification has been limited to Deep Learning. We compare the uncertainty quantification ability of established BCI classifiers using Common Spatial Patterns (CSP-LDA) and Riemannian Geometry (MDRM) to specialized methods in Deep Learning (Deep Ensembles and Direct Uncertainty Quantification) as well as standard Convolutional Neural Networks (CNNs). We found that the overconfidence typically seen in Deep Learning is not a problem in CSP-LDA and MDRM. We found that MDRM is underconfident, which we solved by adding Temperature Scaling (MDRM-T). CSP-LDA and MDRM-T give the best uncertainty estimates, but Deep Ensembles and standard CNNs give the best classifications. We show that all models are able to separate between easy and difficult estimates, so that we can increase the accuracy of a Motor Imagery BCI by rejecting samples that are ambiguous.