Uncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuning

TL;DR

This paper investigates how competing linguistic constraints affect neural model behavior across languages and demonstrates that targeted fine-tuning can reveal hidden linguistic knowledge by re-weighting these constraints.

Contribution

It introduces a method to uncover dormant linguistic knowledge in models by fine-tuning to adjust the influence of competing language constraints.

Findings

Cross-linguistic variation in model behavior was observed.

Targeted fine-tuning can re-weight constraints and reveal hidden linguistic knowledge.

Models need to learn both constraints and their relative importance.

Abstract

A growing body of literature has focused on detailing the linguistic knowledge embedded in large, pretrained language models. Existing work has shown that non-linguistic biases in models can drive model behavior away from linguistic generalizations. We hypothesized that competing linguistic processes within a language, rather than just non-linguistic model biases, could obscure underlying linguistic knowledge. We tested this claim by exploring a single phenomenon in four languages: English, Chinese, Spanish, and Italian. While human behavior has been found to be similar across languages, we find cross-linguistic variation in model behavior. We show that competing processes in a language act as constraints on model behavior and demonstrate that targeted fine-tuning can re-weight the learned constraints, uncovering otherwise dormant linguistic knowledge in models. Our results suggest that models need to learn both the linguistic constraints in a language and their relative ranking, with mismatches in either producing non-human-like behavior.