Are UD Treebanks Getting More Consistent? A Report Card for English UD

TL;DR

This paper evaluates the progress of data consistency and similarity among English UD treebanks, analyzing whether consolidation improves joint training effectiveness and how the datasets compare over time.

Contribution

It provides an empirical assessment of the evolution of consistency and similarity in English UD treebanks, informing the feasibility of joint training.

Findings

Consolidation has improved data consistency.

Joint models still face challenges due to residual inconsistencies.

Progress varies across different versions of UD.

Abstract

Recent efforts to consolidate guidelines and treebanks in the Universal Dependencies project raise the expectation that joint training and dataset comparison is increasingly possible for high-resource languages such as English, which have multiple corpora. Focusing on the two largest UD English treebanks, we examine progress in data consolidation and answer several questions: Are UD English treebanks becoming more internally consistent? Are they becoming more like each other and to what extent? Is joint training a good idea, and if so, since which UD version? Our results indicate that while consolidation has made progress, joint models may still suffer from inconsistencies, which hamper their ability to leverage a larger pool of training data.