EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotyping

TL;DR

This paper introduces EEG-language models trained on clinical reports and EEG data, achieving significant improvements in clinical phenotyping tasks and enabling zero-shot classification and retrieval of neural signals and reports.

Contribution

It pioneers multimodal EEG-language modeling for clinical phenotyping, combining alignment, cropping, segmentation, and multiple instance learning to handle data misalignment.

Findings

Significant improvement over EEG-only models in four clinical evaluations.

First demonstration of zero-shot classification and retrieval in this domain.

Potential for advancing clinical applications with multimodal EEG-language models.

Abstract

Multimodal language modeling has enabled breakthroughs for representation learning, yet remains unexplored in the realm of functional brain data for clinical phenotyping. This paper pioneers EEG-language models (ELMs) trained on clinical reports and 15000 EEGs. We propose to combine multimodal alignment in this novel domain with timeseries cropping and text segmentation, enabling an extension based on multiple instance learning to alleviate misalignment between irrelevant EEG or text segments. Our multimodal models significantly improve over EEG-only models across four clinical evaluations and for the first time enable zero-shot classification as well as retrieval of both neural signals and reports. In sum, these results highlight the potential of ELMs, representing significant progress for clinical applications.