Towards Scalable Handwriting Communication via EEG Decoding and Latent Embedding Integration

TL;DR

This paper presents a novel EEG decoding approach for handwriting recognition, integrating latent embeddings and neural networks to classify handwritten characters with high accuracy, advancing brain-computer interface capabilities.

Contribution

It introduces a method combining CEBRA embeddings and CNNs for EEG-based handwriting classification, achieving 91% accuracy on nine characters, which is a significant step forward.

Findings

Achieved 91% classification accuracy on nine handwritten characters.

Demonstrated the effectiveness of embedding-guided EEG decoding.

Validated the approach with five-fold cross-validation.

Abstract

In recent years, brain-computer interfaces have made advances in decoding various motor-related tasks, including gesture recognition and movement classification, utilizing electroencephalogram (EEG) data. These developments are fundamental in exploring how neural signals can be interpreted to recognize specific physical actions. This study centers on a written alphabet classification task, where we aim to decode EEG signals associated with handwriting. To achieve this, we incorporate hand kinematics to guide the extraction of the consistent embeddings from high-dimensional neural recordings using auxiliary variables (CEBRA). These CEBRA embeddings, along with the EEG, are processed by a parallel convolutional neural network model that extracts features from both data sources simultaneously. The model classifies nine different handwritten characters, including symbols such as exclamation marks and commas, within the alphabet. We evaluate the model using a quantitative five-fold cross-validation approach and explore the structure of the embedding space through visualizations. Our approach achieves a classification accuracy of 91 % for the nine-class task, demonstrating the feasibility of fine-grained handwriting decoding from EEG.