Minor changes make a difference: a case study on the consistency of UD-based dependency parsers

TL;DR

This study investigates how minor input modifications, like replacing numerals, can significantly impact the consistency of UD-based dependency parsers across four languages, highlighting the importance of data augmentation.

Contribution

It demonstrates that small input changes cause large parser output variations and proposes data augmentation as a solution to improve consistency.

Findings

Numeral replacement causes output inconsistencies

Data augmentation reduces parser variability

Effects observed across four languages

Abstract

Many downstream applications are using dependency trees, and are thus relying on dependency parsers producing correct, or at least consistent, output. However, dependency parsers are trained using machine learning, and are therefore susceptible to unwanted inconsistencies due to biases in the training data. This paper explores the effects of such biases in four languages - English, Swedish, Russian, and Ukrainian - though an experiment where we study the effect of replacing numerals in sentences. We show that such seemingly insignificant changes in the input can cause large differences in the output, and suggest that data augmentation can remedy the problems.