Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing

TL;DR

This paper introduces GAQCorpus, a large-scale annotated corpus for theory-based argument quality assessment across multiple online domains, and proposes initial computational models to evaluate argumentation dimensions, advancing NLP's understanding of argument quality.

Contribution

It provides the first large-scale, multi-domain corpus annotated with theory-based AQ scores and develops initial computational models for dimension-specific argument quality assessment.

Findings

Exploiting relations between argumentation dimensions improves model performance.

Large-scale annotation of argument quality is feasible.

Theory-based prediction enhances practical argument quality assessment.

Abstract

Though preceding work in computational argument quality (AQ) mostly focuses on assessing overall AQ, researchers agree that writers would benefit from feedback targeting individual dimensions of argumentation theory. However, a large-scale theory-based corpus and corresponding computational models are missing. We fill this gap by conducting an extensive analysis covering three diverse domains of online argumentative writing and presenting GAQCorpus: the first large-scale English multi-domain (community Q&A forums, debate forums, review forums) corpus annotated with theory-based AQ scores. We then propose the first computational approaches to theory-based assessment, which can serve as strong baselines for future work. We demonstrate the feasibility of large-scale AQ annotation, show that exploiting relations between dimensions yields performance improvements, and explore the synergies between theory-based prediction and practical AQ assessment.