Emergent Semantic Role Understanding in Language Models

TL;DR

This paper investigates whether semantic role understanding in language models emerges during pre-training or requires fine-tuning, finding that pre-training encodes partial role information with more distributed representations at larger scales.

Contribution

It demonstrates that semantic roles are partially encoded during pre-training and that their internal representation becomes more distributed as models scale.

Findings

Pre-trained models contain substantial semantic role information.

Performance improves with scale but does not match fine-tuned models.

Semantic role structure shifts toward more distributed representations as models grow larger.

Abstract

Understanding how linguistic structure emerges in language models is central to interpreting what these systems learn from data and how much supervision they truly require. In particular, semantic role understanding ("who did what to whom") is a core component of meaning representation, yet it remains unclear whether it arises from pre-training alone or depends on task-specific fine-tuning. We study whether semantic role understanding emerges during language model pre-training or requires task-specific fine-tuning. We freeze decoder-only transformers and train linear probes to extract semantic roles, using performance to infer whether role information is already encoded in pre-training or learned during adaptation. Across model scales, we find that frozen representations contain substantial semantic role information, with performance improving but not fully matching fine-tuned models. This indicates partial but incomplete emergence from pre-training alone. We show that semantic role structure emerges from language modeling objectives, but its internal implementation shifts toward more distributed representations as model scale increases.