Inter-rater Agreement on Sentence Formality

TL;DR

This paper investigates inter-rater reliability in assessing sentence formality, analyzes rating difficulties, and aims to develop an automatic formality scoring system for writing style analysis.

Contribution

It provides empirical data on inter-rater agreement for sentence formality and explores factors affecting rating consistency, supporting automatic formality assessment development.

Findings

Good inter-rater agreement achieved

Different rating distributions across sentence categories

Identified bottlenecks in the rating process

Abstract

Formality is one of the most important dimensions of writing style variation. In this study we conducted an inter-rater reliability experiment for assessing sentence formality on a five-point Likert scale, and obtained good agreement results as well as different rating distributions for different sentence categories. We also performed a difficulty analysis to identify the bottlenecks of our rating procedure. Our main objective is to design an automatic scoring mechanism for sentence-level formality, and this study is important for that purpose.