Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better Account for Brain Language Representations?

TL;DR

This study compares prompt-tuning and fine-tuning of pre-trained neural network models to determine which better aligns with human brain language representations, finding prompt-tuning often yields more brain-consistent representations especially for concept-related tasks.

Contribution

The paper demonstrates that prompt-tuning produces neural representations more aligned with brain data than fine-tuning, especially for tasks involving fine-grained concept understanding.

Findings

Prompt-tuning outperforms fine-tuning in neural decoding tasks.

Tasks involving fine-grained concept meaning better decode brain activity.

Brain encodes more detailed concept information than syntactic features.

Abstract

To decipher the algorithm underlying the human brain's language representation, previous work probed brain responses to language input with pre-trained artificial neural network (ANN) models fine-tuned on NLU tasks. However, full fine-tuning generally updates the entire parametric space and distorts pre-trained features, cognitively inconsistent with the brain's robust multi-task learning ability. Prompt-tuning, in contrast, protects pre-trained weights and learns task-specific embeddings to fit a task. Could prompt-tuning generate representations that better account for the brain's language representations than fine-tuning? If so, what kind of NLU task leads a pre-trained model to better decode the information represented in the human brain? We investigate these questions by comparing prompt-tuned and fine-tuned representations in neural decoding, that is predicting the linguistic stimulus from the brain activities evoked by the stimulus. We find that on none of the 10 NLU tasks, full fine-tuning significantly outperforms prompt-tuning in neural decoding, implicating that a more brain-consistent tuning method yields representations that better correlate with brain data. Moreover, we identify that tasks dealing with fine-grained concept meaning yield representations that better decode brain activation patterns than other tasks, especially the syntactic chunking task. This indicates that our brain encodes more fine-grained concept information than shallow syntactic information when representing languages.