Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation

TL;DR

This study investigates how different morphological segmentation models generalize across languages and data variations in low-resource settings, emphasizing the importance of data set characteristics over size for reliable evaluation.

Contribution

It introduces a comprehensive analysis of model generalizability across multiple languages and data conditions, highlighting key data characteristics affecting performance.

Findings

Model generalization varies with data set characteristics.

Data set size is less influential than morpheme overlap.

Random sampling of data sets improves evaluation reliability.

Abstract

Common designs of model evaluation typically focus on monolingual settings, where different models are compared according to their performance on a single data set that is assumed to be representative of all possible data for the task at hand. While this may be reasonable for a large data set, this assumption is difficult to maintain in low-resource scenarios, where artifacts of the data collection can yield data sets that are outliers, potentially making conclusions about model performance coincidental. To address these concerns, we investigate model generalizability in crosslinguistic low-resource scenarios. Using morphological segmentation as the test case, we compare three broad classes of models with different parameterizations, taking data from 11 languages across 6 language families. In each experimental setting, we evaluate all models on a first data set, then examine their performance consistency when introducing new randomly sampled data sets with the same size and when applying the trained models to unseen test sets of varying sizes. The results demonstrate that the extent of model generalization depends on the characteristics of the data set, and does not necessarily rely heavily on the data set size. Among the characteristics that we studied, the ratio of morpheme overlap and that of the average number of morphemes per word between the training and test sets are the two most prominent factors. Our findings suggest that future work should adopt random sampling to construct data sets with different sizes in order to make more responsible claims about model evaluation.