Challenges in Trustworthy Human Evaluation of Chatbots

TL;DR

This paper examines the difficulties in ensuring high-quality human annotations for chatbot evaluation platforms, highlighting how even a small percentage of poor-quality votes can significantly distort model rankings and discussing open challenges in this area.

Contribution

It identifies key sources of annotation quality issues in open leaderboard platforms and quantifies their impact on model ranking reliability.

Findings

10% poor annotations can shift rankings by 5 places

Malicious and apathetic annotators significantly affect leaderboard accuracy

Addressing annotation quality is crucial for trustworthy chatbot evaluation

Abstract

Open community-driven platforms like Chatbot Arena that collect user preference data from site visitors have gained a reputation as one of the most trustworthy publicly available benchmarks for LLM performance. While now standard, it is tricky to implement effective guardrails to collect high-quality annotations from humans. In this paper, we demonstrate that three sources of bad annotations, both malicious and otherwise, can corrupt the reliability of open leaderboard rankings. In particular, we show that only 10\% of poor quality votes by apathetic (site visitors not appropriately incentivized to give correct votes) or adversarial (bad actors seeking to inflate the ranking of a target model) annotators can change the rankings of models by up to 5 places on the leaderboard. Finally, we discuss open challenges in ensuring high-quality human annotations.