Tuning In to Neural Encoding: Linking Human Brain and Artificial Supervised Representations of Language

TL;DR

This study explores how prompt-tuning of supervised Transformer models on Chinese language tasks enhances neural encoding of brain responses, revealing task-specific influences on language representation alignment.

Contribution

It introduces prompt-tuning for neural encoding in Chinese, demonstrating its superiority over traditional fine-tuning and identifying key tasks that improve brain response prediction.

Findings

Prompt-tuning outperforms fine-tuning in neural response prediction.

Tasks involving detailed concept and entity processing are most effective.

The proportion of tuned parameters significantly affects encoding performance.

Abstract

To understand the algorithm that supports the human brain's language representation, previous research has attempted to predict neural responses to linguistic stimuli using embeddings generated by artificial neural networks (ANNs), a process known as neural encoding. However, most of these studies have focused on probing neural representations of Germanic languages, such as English, with unsupervised ANNs. In this paper, we propose to bridge the gap between human brain and supervised ANN representations of the Chinese language. Specifically, we investigate how task tuning influences a pretained Transformer for neural encoding and which tasks lead to the best encoding performances. We generate supervised representations on eight Natural Language Understanding (NLU) tasks using prompt-tuning, a technique that is seldom explored in neural encoding for language. We demonstrate that prompt-tuning yields representations that better predict neural responses to Chinese stimuli than traditional fine-tuning on four tasks. Furthermore, we discover that tasks that require a fine-grained processing of concepts and entities lead to representations that are most predictive of brain activation patterns. Additionally, we reveal that the proportion of tuned parameters highly influences the neural encoding performance of fine-tuned models. Overall, our experimental findings could help us better understand the relationship between supervised artificial and brain language representations.