Argumentation Element Annotation Modeling using XLNet

TL;DR

This paper demonstrates that fine-tuned XLNet models effectively annotate argumentative elements in essays, outperforming previous methods and even surpassing human agreement levels in some cases, across multiple datasets.

Contribution

The study introduces a transformer-based approach using XLNet for argument element annotation, showing its robustness across diverse datasets and annotation schemes.

Findings

XLNet achieves high annotation accuracy, surpassing human agreement in some cases.

The model effectively handles long essays and diverse annotation schemes.

Insights into relationships between annotation tags were revealed.

Abstract

This study demonstrates the effectiveness of XLNet, a transformer-based language model, for annotating argumentative elements in persuasive essays. XLNet's architecture incorporates a recurrent mechanism that allows it to model long-term dependencies in lengthy texts. Fine-tuned XLNet models were applied to three datasets annotated with different schemes - a proprietary dataset using the Annotations for Revisions and Reflections on Writing (ARROW) scheme, the PERSUADE corpus, and the Argument Annotated Essays (AAE) dataset. The XLNet models achieved strong performance across all datasets, even surpassing human agreement levels in some cases. This shows XLNet capably handles diverse annotation schemes and lengthy essays. Comparisons between the model outputs on different datasets also revealed insights into the relationships between the annotation tags. Overall, XLNet's strong performance on modeling argumentative structures across diverse datasets highlights its suitability for providing automated feedback on essay organization.