E^2-LLM: Bridging Neural Signals and Interpretable Affective Analysis

TL;DR

E^2-LLM is a novel framework that combines EEG signals with large language models to improve emotion recognition, interpretability, and zero-shot reasoning in affective computing.

Contribution

This work introduces the first MLLM framework for EEG-based emotion analysis, integrating neural signals with LLMs through a multi-stage training pipeline and demonstrating enhanced performance and interpretability.

Findings

Achieves high accuracy in emotion classification across seven categories.

Larger model variants show better zero-shot generalization and reasoning.

Establishes a new paradigm for physiological signal and LLM integration.

Abstract

Emotion recognition from electroencephalography (EEG) signals remains challenging due to high inter-subject variability, limited labeled data, and the lack of interpretable reasoning in existing approaches. While recent multimodal large language models (MLLMs) have advanced emotion analysis, they have not been adapted to handle the unique spatiotemporal characteristics of neural signals. We present E^2-LLM (EEG-to-Emotion Large Language Model), the first MLLM framework for interpretable emotion analysis from EEG. E^2-LLM integrates a pretrained EEG encoder with Qwen-based LLMs through learnable projection layers, employing a multi-stage training pipeline that encompasses emotion-discriminative pretraining, cross-modal alignment, and instruction tuning with chain-of-thought reasoning. We design a comprehensive evaluation protocol covering basic emotion prediction, multi-task reasoning, and zero-shot scenario understanding. Experiments on the dataset across seven emotion categories demonstrate that E^2-LLM achieves excellent performance on emotion classification, with larger variants showing enhanced reliability and superior zero-shot generalization to complex reasoning scenarios. Our work establishes a new paradigm combining physiological signals with LLM reasoning capabilities, showing that model scaling improves both recognition accuracy and interpretable emotional understanding in affective computing.