Human-Centered Design Recommendations for LLM-as-a-Judge

TL;DR

This paper explores how to effectively integrate human input into LLM-based evaluation systems to improve trust, reliability, and alignment with human judgment, through a user-centered design approach.

Contribution

It introduces EvaluLLM, a design framework that combines human expertise with LLM evaluation, and provides practical recommendations based on expert interviews.

Findings

Human input is essential for aligning LLM evaluations with user expectations.

Design recommendations improve the effectiveness of human-assisted LLM evaluation systems.

Expert interviews highlight key challenges and solutions in developing evaluation criteria.

Abstract

Traditional reference-based metrics, such as BLEU and ROUGE, are less effective for assessing outputs from Large Language Models (LLMs) that produce highly creative or superior-quality text, or in situations where reference outputs are unavailable. While human evaluation remains an option, it is costly and difficult to scale. Recent work using LLMs as evaluators (LLM-as-a-judge) is promising, but trust and reliability remain a significant concern. Integrating human input is crucial to ensure criteria used to evaluate are aligned with the human's intent, and evaluations are robust and consistent. This paper presents a user study of a design exploration called EvaluLLM, that enables users to leverage LLMs as customizable judges, promoting human involvement to balance trust and cost-saving potential with caution. Through interviews with eight domain experts, we identified the need for assistance in developing effective evaluation criteria aligning the LLM-as-a-judge with practitioners' preferences and expectations. We offer findings and design recommendations to optimize human-assisted LLM-as-judge systems.