Parser agreement and disagreement in L2 Korean UD: Implications for human-in-the-loop annotation

TL;DR

This paper explores a semi-automatic approach for annotating L2 Korean syntax by using parser agreement as a proxy for correctness, showing promising results and identifying key challenges.

Contribution

It introduces a simplified human-in-the-loop workflow leveraging parser agreement for L2 Korean UD annotation, validated against human judgments.

Findings

Strong correlation between parser agreement and human judgments.

Parser disagreements cluster in predictable linguistic domains.

Some disagreements are due to fundamental representational challenges.

Abstract

We propose a simplified human-in-the-loop workflow for second language (L2) Korean morphosyntactic annotation by leveraging agreement between two domain-adapted parsers. We first evaluate whether parser agreement can serve as a proxy for annotation correctness by comparing it with independent human judgments. The results show strong correspondence between parser and human judgments, supporting the feasibility of semi-automatic L2-Korean UD annotation. Further analysis demonstrates that parser disagreements cluster in linguistically predictable domains such as grammatical-relation distinctions and clause-boundary ambiguity. While many disagreement cases are tractable for iterative model refinement, others reflect deeper representational challenges inherent in parsing and tagging L2-Korean corpora.