Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications

TL;DR

This paper evaluates the use of large language models for complex search annotation tasks, highlighting their limitations and proposing a human-in-the-loop approach to improve reliability and reduce human effort.

Contribution

It introduces a systematic assessment of LLMs in nuanced annotation tasks and proposes a simple HITL workflow to enhance annotation quality efficiently.

Findings

LLMs struggle with subjective and fine-grained annotations.

The proposed HITL workflow reduces human effort by up to 45%.

LLM predictions are often inconsistent and sensitive to prompts.

Abstract

Despite growing interest in using large language models (LLMs) to automate annotation, their effectiveness in complex, nuanced, and multi-dimensional labelling tasks remains relatively underexplored. This study focuses on annotation for the search clarification task, leveraging a high-quality, multi-dimensional dataset that includes five distinct fine-grained annotation subtasks. Although LLMs have shown impressive capabilities in general settings, our study reveals that even state-of-the-art models struggle to replicate human-level performance in subjective or fine-grained evaluation tasks. Through a systematic assessment, we demonstrate that LLM predictions are often inconsistent, poorly calibrated, and highly sensitive to prompt variations. To address these limitations, we propose a simple yet effective human-in-the-loop (HITL) workflow that uses confidence thresholds and inter-model disagreement to selectively involve human review. Our findings show that this lightweight intervention significantly improves annotation reliability while reducing human effort by up to 45%, offering a relatively scalable and cost-effective yet accurate path forward for deploying LLMs in real-world evaluation settings.