Aligning Brain Activity with Advanced Transformer Models: Exploring the Role of Punctuation in Semantic Processing

TL;DR

This study explores how advanced transformer models align with neural brain activity during language comprehension, highlighting the role of punctuation and demonstrating RoBERTa's superior neural alignment compared to other models.

Contribution

It introduces an innovative evaluation of transformer models' neural alignment and reveals punctuation's impact on model performance and brain activity correlation.

Findings

RoBERTa shows the closest neural alignment among models.

Removing punctuation improves BERT's accuracy and neural alignment.

Punctuation significantly influences semantic processing in neural networks.

Abstract

This research examines the congruence between neural activity and advanced transformer models, emphasizing the semantic significance of punctuation in text understanding. Utilizing an innovative approach originally proposed by Toneva and Wehbe, we evaluate four advanced transformer models RoBERTa, DistiliBERT, ALBERT, and ELECTRA against neural activity data. Our findings indicate that RoBERTa exhibits the closest alignment with neural activity, surpassing BERT in accuracy. Furthermore, we investigate the impact of punctuation removal on model performance and neural alignment, revealing that BERT's accuracy enhances in the absence of punctuation. This study contributes to the comprehension of how neural networks represent language and the influence of punctuation on semantic processing within the human brain.