Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models

TL;DR

This paper introduces a novel method using linguistic minimal pairs to analyze and compare the internal linguistic representations of large language models across multiple languages and categories.

Contribution

It presents a large-scale analysis framework that measures linguistic similarity in LLMs using minimal pairs, revealing insights into their linguistic knowledge and cross-lingual properties.

Findings

Linguistic similarity correlates with training data exposure.

Strong alignment with fine-grained linguistic categories.

Weak correlation between linguistic and semantic similarity.

Abstract

We introduce a novel analysis that leverages linguistic minimal pairs to probe the internal linguistic representations of Large Language Models (LLMs). By measuring the similarity between LLM activation differences across minimal pairs, we quantify the and gain insight into the linguistic knowledge captured by LLMs. Our large-scale experiments, spanning 100+ LLMs and 150k minimal pairs in three languages, reveal properties of linguistic similarity from four key aspects: consistency across LLMs, relation to theoretical categorizations, dependency to semantic context, and cross-lingual alignment of relevant phenomena. Our findings suggest that 1) linguistic similarity is significantly influenced by training data exposure, leading to higher cross-LLM agreement in higher-resource languages. 2) Linguistic similarity strongly aligns with fine-grained theoretical linguistic categories but weakly with broader ones. 3) Linguistic similarity shows a weak correlation with semantic similarity, showing its context-dependent nature. 4) LLMs exhibit limited cross-lingual alignment in their understanding of relevant linguistic phenomena. This work demonstrates the potential of minimal pairs as a window into the neural representations of language in LLMs, shedding light on the relationship between LLMs and linguistic theory. Codes and data are available at https://github.com/ChenDelong1999/Linguistic-Similarity