Towards a Perspectivist Turn in Argument Quality Assessment

TL;DR

This paper explores the subjective nature of argument quality assessment in NLP, emphasizing the importance of perspectives and annotator diversity, and provides a systematic review of relevant datasets to support perspectivist modeling.

Contribution

It introduces a multi-layered categorization of argument quality datasets and highlights the significance of annotator information for developing perspectivist models.

Findings

Annotated datasets vary in quality dimensions and annotator details.

A taxonomy improves dataset comparability and interoperability.

Controlled annotator selection impacts perspectivist model development.

Abstract

The assessment of argument quality depends on well-established logical, rhetorical, and dialectical properties that are unavoidably subjective: multiple valid assessments may exist, there is no unequivocal ground truth. This aligns with recent paths in machine learning, which embrace the co-existence of different perspectives. However, this potential remains largely unexplored in NLP research on argument quality. One crucial reason seems to be the yet unexplored availability of suitable datasets. We fill this gap by conducting a systematic review of argument quality datasets. We assign them to a multi-layered categorization targeting two aspects: (a) What has been annotated: we collect the quality dimensions covered in datasets and consolidate them in an overarching taxonomy, increasing dataset comparability and interoperability. (b) Who annotated: we survey what information is given about annotators, enabling perspectivist research and grounding our recommendations for future actions. To this end, we discuss datasets suitable for developing perspectivist models (i.e., those containing individual, non-aggregated annotations), and we showcase the importance of a controlled selection of annotators in a pilot study.