SI-SD: Sleep Interpreter through awake-guided cross-subject Semantic Decoding

TL;DR

This paper introduces SI-SD, a novel framework for decoding semantic content from sleep EEG data, leveraging a new dataset and alignment techniques to improve accuracy across sleep stages and subjects.

Contribution

The study presents a comprehensive EEG dataset and a novel alignment-based model, significantly advancing sleep semantic decoding and cross-subject generalization.

Findings

Achieved over 24% top-1 accuracy in sleep semantic decoding on unseen subjects.

Fine-tuning improved accuracy to over 30% across sleep stages.

Decoding performance increased to 40% when considering Slow Oscillation events.

Abstract

Understanding semantic content from brain activity during sleep represents a major goal in neuroscience. While studies in rodents have shown spontaneous neural reactivation of memories during sleep, capturing the semantic content of human sleep poses a significant challenge due to the absence of well-annotated sleep datasets and the substantial differences in neural patterns between wakefulness and sleep. To address these challenges, we designed a novel cognitive neuroscience experiment and collected a comprehensive, well-annotated electroencephalography (EEG) dataset from 134 subjects during both wakefulness and sleep. Leveraging this benchmark dataset, we developed SI-SD that enhances sleep semantic decoding through the position-wise alignment of neural latent sequence between wakefulness and sleep. In the 15-way classification task, our model achieves 24.12% and 21.39% top-1 accuracy on unseen subjects for NREM 2/3 and REM sleep, respectively, surpassing all other baselines. With additional fine-tuning, decoding performance improves to 30.32% and 31.65%, respectively. Besides, inspired by previous neuroscientific findings, we systematically analyze how the "Slow Oscillation" event impacts decoding performance in NREM 2/3 sleep -- decoding performance on unseen subjects further improves to 40.02%. Together, our findings and methodologies contribute to a promising neuro-AI framework for decoding brain activity during sleep.