Meaning-infused grammar: Gradient Acceptability Shapes the Geometric Representations of Constructions in LLMs

TL;DR

This paper demonstrates that Large Language Models encode graded, meaning-infused representations of language constructions, with their internal geometric separability reflecting human-like preference strengths for different sentence structures.

Contribution

It provides empirical evidence that LLMs learn rich, graded, and meaning-infused representations of constructions, supporting the constructionist view of language.

Findings

Representation separability varies systematically with preference strength.

Prototypical exemplars are more geometrically distinct in activation space.

Results support geometric measures as tools for analyzing LLM representations.

Abstract

The usage-based constructionist (UCx) approach to language posits that language comprises a network of learned form-meaning pairings (constructions) whose use is largely determined by their meanings or functions, requiring them to be graded and probabilistic. This study investigates whether the internal representations in Large Language Models (LLMs) reflect the proposed function-infused gradience. We analyze representations of the English Double Object (DO) and Prepositional Object (PO) constructions in Pythia-$1.4$B, using a dataset of $5000$ sentence pairs systematically varied by human-rated preference strength for DO or PO. Geometric analyses show that the separability between the two constructions' representations, as measured by energy distance or Jensen-Shannon divergence, is systematically modulated by gradient preference strength, which depends on lexical and functional properties of sentences. That is, more prototypical exemplars of each construction occupy more distinct regions in activation space, compared to sentences that could have equally well have occured in either construction. These results provide evidence that LLMs learn rich, meaning-infused, graded representations of constructions and offer support for geometric measures for representations in LLMs.