Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies

TL;DR

This paper introduces curated, multilingual datasets based on Blackbird Language Matrices to evaluate large language models' understanding of verb alternations through controlled linguistic puzzles.

Contribution

It presents novel, systematic datasets and data augmentation strategies for probing cross-sentence verb alternation knowledge in multiple languages.

Findings

Baseline results show datasets are effective diagnostic tools.

Datasets cover multiple languages and verb alternation types.

Data augmentation improves dataset diversity.

Abstract

Large language models (LLMs) have shown remarkable performance across various sentence-based linguistic phenomena, yet their ability to capture cross-sentence paradigmatic patterns, such as verb alternations, remains underexplored. In this work, we present curated paradigm-based datasets for four languages, designed to probe systematic cross-sentence knowledge of verb alternations (change-of-state and object-drop constructions in English, German and Italian, and Hebrew binyanim). The datasets comprise thousands of the Blackbird Language Matrices (BLMs) problems. The BLM task -- an RPM/ARC-like task devised specifically for language -- is a controlled linguistic puzzle where models must select the sentence that completes a pattern according to syntactic and semantic rules. We introduce three types of templates varying in complexity and apply linguistically-informed data augmentation strategies across synthetic and natural data. We provide simple baseline performance results across English, Italian, German, and Hebrew, that demonstrate the diagnostic usefulness of the datasets.