Linking artificial and human neural representations of language

TL;DR

This study compares pre-trained language models and NLU tasks to understand what linguistic information is robustly represented in the human brain during sentence comprehension, revealing that syntax-light representations improve brain decoding.

Contribution

It demonstrates that fine-tuning BERT on certain NLU tasks does not enhance brain decoding, highlighting the importance of syntax-light representations for neural alignment.

Findings

Syntax-light tasks improve brain decoding performance.

Fine-tuning on NLU tasks does not significantly enhance neural decoding.

Limits on decoding fine-grained syntactic information from fMRI.

Abstract

What information from an act of sentence understanding is robustly represented in the human brain? We investigate this question by comparing sentence encoding models on a brain decoding task, where the sentence that an experimental participant has seen must be predicted from the fMRI signal evoked by the sentence. We take a pre-trained BERT architecture as a baseline sentence encoding model and fine-tune it on a variety of natural language understanding (NLU) tasks, asking which lead to improvements in brain-decoding performance. We find that none of the sentence encoding tasks tested yield significant increases in brain decoding performance. Through further task ablations and representational analyses, we find that tasks which produce syntax-light representations yield significant improvements in brain decoding performance. Our results constrain the space of NLU models that could best account for human neural representations of language, but also suggest limits on the possibility of decoding fine-grained syntactic information from fMRI human neuroimaging.