SQUINKY! A Corpus of Sentence-level Formality, Informativeness, and Implicature

TL;DR

This paper introduces a large, human-annotated corpus of 7,032 sentences rated for formality, informativeness, and implicature, enabling better analysis and automatic scoring of stylistic features in text.

Contribution

The creation and validation of the largest sentence-level corpus annotated for formality, informativeness, and implicature, with analysis of annotation reliability and linguistic correlations.

Findings

High correlation in ratings across experiments for formality and informativeness.

Significant genre-wise variation in stylistic ratings.

Compatibility of the corpus with automatic stylistic scoring methods.

Abstract

We introduce a corpus of 7,032 sentences rated by human annotators for formality, informativeness, and implicature on a 1-7 scale. The corpus was annotated using Amazon Mechanical Turk. Reliability in the obtained judgments was examined by comparing mean ratings across two MTurk experiments, and correlation with pilot annotations (on sentence formality) conducted in a more controlled setting. Despite the subjectivity and inherent difficulty of the annotation task, correlations between mean ratings were quite encouraging, especially on formality and informativeness. We further explored correlation between the three linguistic variables, genre-wise variation of ratings and correlations within genres, compatibility with automatic stylistic scoring, and sentential make-up of a document in terms of style. To date, our corpus is the largest sentence-level annotated corpus released for formality, informativeness, and implicature.