Inter-Rater Agreement Study on Readability Assessment in Bengali

TL;DR

This study evaluates inter-rater agreement on Bengali readability assessment using a 1-7 scale, resulting in a new annotated dataset and insights into annotation consistency.

Contribution

It introduces a Bengali readability dataset with annotated ratings and analyzes inter-rater agreement, which is novel for this language and task.

Findings

Moderate to fair agreement among annotators

Created a ground truth dataset for Bengali readability

Insights into annotation consistency and variability

Abstract

An inter-rater agreement study is performed for readability assessment in Bengali. A 1-7 rating scale was used to indicate different levels of readability. We obtained moderate to fair agreement among seven independent annotators on 30 text passages written by four eminent Bengali authors. As a by product of our study, we obtained a readability-annotated ground truth dataset in Bengali. .