Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement

TL;DR

This paper investigates how large language models detect offensive language, especially in cases of annotation disagreement, revealing their tendency to be overconfident in ambiguous situations and how training with disagreement samples can improve performance.

Contribution

The study systematically evaluates LLMs' performance on offensive language detection across different levels of annotation agreement and explores methods to improve their alignment with human judgments.

Findings

LLMs struggle with low-agreement samples.

LLMs tend to be overconfident in ambiguous cases.

Training with disagreement samples enhances detection accuracy and alignment.

Abstract

Large Language Models (LLMs) have become essential for offensive language detection, yet their ability to handle annotation disagreement remains underexplored. Disagreement samples, which arise from subjective interpretations, pose a unique challenge due to their ambiguous nature. Understanding how LLMs process these cases, particularly their confidence levels, can offer insight into their alignment with human annotators. This study systematically evaluates the performance of multiple LLMs in detecting offensive language at varying levels of annotation agreement. We analyze binary classification accuracy, examine the relationship between model confidence and human disagreement, and explore how disagreement samples influence model decision-making during few-shot learning and instruction fine-tuning. Our findings reveal that LLMs struggle with low-agreement samples, often exhibiting overconfidence in these ambiguous cases. However, utilizing disagreement samples in training improves both detection accuracy and model alignment with human judgment. These insights provide a foundation for enhancing LLM-based offensive language detection in real-world moderation tasks.