DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errors

TL;DR

This paper introduces DaLA, a comprehensive Danish linguistic acceptability benchmark created by systematically corrupting correct sentences to evaluate language models' ability to detect errors, thus providing a more rigorous assessment tool.

Contribution

The paper develops a novel set of corruption functions for Danish, creating a more extensive and challenging benchmark for linguistic acceptability evaluation of language models.

Findings

LLMs perform worse on DaLA than existing benchmarks.

DaLA's corruption methods increase task difficulty.

The benchmark better distinguishes model performance.

Abstract

We present an enhanced benchmark for evaluating linguistic acceptability in Danish. We first analyze the most common errors found in written Danish. Based on this analysis, we introduce a set of fourteen corruption functions that generate incorrect sentences by systematically introducing errors into existing correct Danish sentences. To ensure the accuracy of these corruptions, we assess their validity using both manual and automatic methods. The results are then used as a benchmark for evaluating Large Language Models on a linguistic acceptability judgement task. Our findings demonstrate that this extension is both broader and more comprehensive than the current state of the art. By incorporating a greater variety of corruption types, our benchmark provides a more rigorous assessment of linguistic acceptability, increasing task difficulty, as evidenced by the lower performance of LLMs on our benchmark compared to existing ones. Our results also suggest that our benchmark has a higher discriminatory power which allows to better distinguish well-performing models from low-performing ones.