The Fragility of Multi-Treebank Parsing Evaluation

TL;DR

This paper investigates how the choice of treebanks impacts parser evaluation, revealing that evaluations on different subsets can lead to inconsistent conclusions and highlighting the importance of careful treebank selection.

Contribution

It systematically analyzes the variability in parser evaluation results caused by different treebank subsets and proposes methods to identify potentially harmful selection strategies.

Findings

Evaluation results vary significantly across different treebank subsets.

Randomly selected subsets can produce inconsistent rankings of parsers.

Guidelines for better treebank selection can help mitigate evaluation biases.

Abstract

Treebank selection for parsing evaluation and the spurious effects that might arise from a biased choice have not been explored in detail. This paper studies how evaluating on a single subset of treebanks can lead to weak conclusions. First, we take a few contrasting parsers, and run them on subsets of treebanks proposed in previous work, whose use was justified (or not) on criteria such as typology or data scarcity. Second, we run a large-scale version of this experiment, create vast amounts of random subsets of treebanks, and compare on them many parsers whose scores are available. The results show substantial variability across subsets and that although establishing guidelines for good treebank selection is hard, it is possible to detect potentially harmful strategies.