MEGnifying Emotion: Sentiment Analysis from Annotated Brain Data

TL;DR

This paper presents a novel approach to decode sentiment from MEG brain data by annotating audio with sentiment labels using pre-trained models and aligning these with brain recordings, demonstrating improved accuracy.

Contribution

The study introduces a new method for annotating brain data with sentiment labels and aligns it with audio to train brain-to-sentiment models, filling a gap in existing datasets.

Findings

Improved balanced accuracy in brain-to-sentiment decoding.

Effective use of pre-trained sentiment models for brain data annotation.

Proof-of-concept for sentiment decoding from non-invasive brain recordings.

Abstract

Decoding emotion from brain activity could unlock a deeper understanding of the human experience. While a number of existing datasets align brain data with speech and with speech transcripts, no datasets have annotated brain data with sentiment. To bridge this gap, we explore the use of pre-trained Text-to-Sentiment models to annotate non invasive brain recordings, acquired using magnetoencephalography (MEG), while participants listened to audiobooks. Having annotated the text, we employ force-alignment of the text and audio to align our sentiment labels with the brain recordings. It is straightforward then to train Brainto-Sentiment models on these data. Experimental results show an improvement in balanced accuracy for Brain-to-Sentiment compared to baseline, supporting the proposed approach as a proof-of-concept for leveraging existing MEG datasets and learning to decode sentiment directly from the brain.