Inter-rater Agreement on Sentence Formality
Shibamouli Lahiri, Xiaofei Lu

TL;DR
This paper investigates inter-rater reliability in assessing sentence formality, analyzes rating difficulties, and aims to develop an automatic formality scoring system for writing style analysis.
Contribution
It provides empirical data on inter-rater agreement for sentence formality and explores factors affecting rating consistency, supporting automatic formality assessment development.
Findings
Good inter-rater agreement achieved
Different rating distributions across sentence categories
Identified bottlenecks in the rating process
Abstract
Formality is one of the most important dimensions of writing style variation. In this study we conducted an inter-rater reliability experiment for assessing sentence formality on a five-point Likert scale, and obtained good agreement results as well as different rating distributions for different sentence categories. We also performed a difficulty analysis to identify the bottlenecks of our rating procedure. Our main objective is to design an automatic scoring mechanism for sentence-level formality, and this study is important for that purpose.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Software Engineering Research
