Semantic-aware Contrastive Learning for Electroencephalography-to-Text Generation with Curriculum Learning

TL;DR

This paper introduces a curriculum semantic-aware contrastive learning method for EEG-to-Text generation, effectively aligning EEG representations with semantic content, leading to improved performance and state-of-the-art results across multiple settings.

Contribution

It proposes a novel curriculum semantic-aware contrastive learning strategy that enhances EEG-to-Text generation by reducing subject-dependent to semantic-dependent representation discrepancies.

Findings

Achieves state-of-the-art results on the ZuCo benchmark.

Improves performance in single-subject and low-resource settings.

Demonstrates robustness and generalizability in zero-shot scenarios.

Abstract

Electroencephalography-to-Text generation (EEG-to-Text), which aims to directly generate natural text from EEG signals has drawn increasing attention in recent years due to the enormous potential for Brain-computer interfaces (BCIs). However, the remarkable discrepancy between the subject-dependent EEG representation and the semantic-dependent text representation poses a great challenge to this task. To mitigate this challenge, we devise a Curriculum Semantic-aware Contrastive Learning strategy (C-SCL), which effectively re-calibrates the subject-dependent EEG representation to the semantic-dependent EEG representation, thus reducing the discrepancy. Specifically, our C-SCL pulls semantically similar EEG representations together while pushing apart dissimilar ones. Besides, in order to introduce more meaningful contrastive pairs, we carefully employ curriculum learning to not only craft meaningful contrastive pairs but also make the learning progressively. We conduct extensive experiments on the ZuCo benchmark and our method combined with diverse models and architectures shows stable improvements across three types of metrics while achieving the new state-of-the-art. Further investigation proves not only its superiority in both the single-subject and low-resource settings but also its robust generalizability in the zero-shot setting.