A Large-scale Dataset for Argument Quality Ranking: Construction and Analysis

TL;DR

This paper introduces the largest dataset for argument quality ranking, along with an analysis of annotation methods and a neural model that outperforms existing baselines.

Contribution

It provides the largest annotated corpus for argument quality, evaluates annotation aggregation methods, and proposes a neural ranking model.

Findings

Neural method outperforms baseline models

Largest dataset for argument quality to date

Effective annotation aggregation strategies identified

Abstract

Identifying the quality of free-text arguments has become an important task in the rapidly expanding field of computational argumentation. In this work, we explore the challenging task of argument quality ranking. To this end, we created a corpus of 30,497 arguments carefully annotated for point-wise quality, released as part of this work. To the best of our knowledge, this is the largest dataset annotated for point-wise argument quality, larger by a factor of five than previously released datasets. Moreover, we address the core issue of inducing a labeled score from crowd annotations by performing a comprehensive evaluation of different approaches to this problem. In addition, we analyze the quality dimensions that characterize this dataset. Finally, we present a neural method for argument quality ranking, which outperforms several baselines on our own dataset, as well as previous methods published for another dataset.