Are LLMs Ready to Replace Bangla Annotators?

TL;DR

This paper evaluates the reliability of large language models as zero-shot annotators for Bangla hate speech detection, revealing biases and instability that challenge their use in sensitive, low-resource language tasks.

Contribution

It systematically benchmarks 17 LLMs for Bangla hate speech annotation, uncovering biases and instability, and challenges assumptions about model size correlating with annotation quality.

Findings

Larger models do not always produce better annotations.

Smaller, task-aligned models can be more consistent.

LLMs exhibit bias and instability in sensitive tasks.

Abstract

Large Language Models (LLMs) are increasingly used as automated annotators to scale dataset creation, yet their reliability as unbiased annotators--especially for low-resource and identity-sensitive settings--remains poorly understood. In this work, we study the behavior of LLMs as zero-shot annotators for Bangla hate speech, a task where even human agreement is challenging, and annotator bias can have serious downstream consequences. We conduct a systematic benchmark of 17 LLMs using a unified evaluation framework. Our analysis uncovers annotator bias and substantial instability in model judgments. Surprisingly, increased model scale does not guarantee improved annotation quality--smaller, more task-aligned models frequently exhibit more consistent behavior than their larger counterparts. These results highlight important limitations of current LLMs for sensitive annotation tasks in low-resource languages and underscore the need for careful evaluation before deployment.