ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models

TL;DR

This paper advocates for a multidisciplinary, reliable, and scalable human evaluation framework for large language models, emphasizing usability, cognitive biases, and effective test sets.

Contribution

It introduces ConSiDERS, a comprehensive six-pillar framework to improve human evaluation of generative LLMs, integrating insights from multiple disciplines.

Findings

Highlighting cognitive biases affecting evaluations

Proposing a six-pillar evaluation framework

Addressing scalability challenges in human assessment

Abstract

In this position paper, we argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking that draws upon insights from disciplines such as user experience research and human behavioral psychology to ensure that the experimental design and results are reliable. The conclusions from these evaluations, thus, must consider factors such as usability, aesthetics, and cognitive biases. We highlight how cognitive biases can conflate fluent information and truthfulness, and how cognitive uncertainty affects the reliability of rating scores such as Likert. Furthermore, the evaluation should differentiate the capabilities and weaknesses of increasingly powerful large language models -- which requires effective test sets. The scalability of human evaluation is also crucial to wider adoption. Hence, to design an effective human evaluation system in the age of generative NLP, we propose the ConSiDERS-The-Human evaluation framework consisting of 6 pillars -- Consistency, Scoring Criteria, Differentiating, User Experience, Responsible, and Scalability.