Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italian

TL;DR

This paper investigates how natural and synthetic structured data affect language models' understanding of passive verb forms in French and Italian, highlighting the importance of natural data for robust linguistic knowledge.

Contribution

It demonstrates that models trained on natural data generalize better to real language, emphasizing the value of natural datasets in linguistic evaluation of LLMs.

Findings

Models trained on synthetic data perform well on synthetic tests but poorly on natural sentences.

Models trained on natural data perform robustly across both natural and synthetic tests.

Natural data enhances models' ability to capture abstract linguistic patterns.

Abstract

This study compares the impact of natural and synthetic data on training and evaluating large language models (LLMs), using the case of passive verb alternation in French and Italian. We use Blackbird Language Matrices (BLMs), structured datasets designed to probe linguistic knowledge of underlying patterns across sentence sets. We compare structured templates instantiated with natural sentences extracted from Universal Dependencies to structured templates of synthetic sentences. Experiments show that while models achieve ceiling performance when trained and tested on synthetic datasets, they do not reliably generalize to natural sentences. In contrast, models trained on natural data exhibit robust performance across both natural and synthetic test suites, demonstrating their superior ability to capture abstract linguistic patterns. These results corroborate the value of natural data and of structured set ups in linguistic evaluation for probing LLMs' syntactic and semantic knowledge.