Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation

TL;DR

This paper introduces MAJ-EVAL, a multi-agent framework that uses diverse LLM-based evaluators to simulate human multi-dimensional evaluation, improving alignment with human judgments in NLP assessments.

Contribution

The paper presents a novel framework for automatically creating diverse evaluator personas and multi-agent debates, enhancing the generalizability and accuracy of LLM-based evaluations.

Findings

MAJ-EVAL produces evaluations more aligned with human experts than traditional metrics.

The framework is effective in both educational and medical NLP domains.

Multi-agent debates improve the robustness of automated evaluations.

Abstract

Nearly all human work is collaborative; thus, the evaluation of real-world NLP applications often requires multiple dimensions that align with diverse human perspectives. As real human evaluator resources are often scarce and costly, the emerging "LLM-as-a-judge" paradigm sheds light on a promising approach to leverage LLM agents to believably simulate human evaluators. Yet, to date, existing LLM-as-a-judge approaches face two limitations: persona descriptions of agents are often arbitrarily designed, and the frameworks are not generalizable to other tasks. To address these challenges, we propose MAJ-EVAL, a Multi-Agent-as-Judge evaluation framework that can automatically construct multiple evaluator personas with distinct dimensions from relevant text documents (e.g., research papers), instantiate LLM agents with the personas, and engage in-group debates with multi-agents to Generate multi-dimensional feedback. Our evaluation experiments in both the educational and medical domains demonstrate that MAJ-EVAL can generate evaluation results that better align with human experts' ratings compared with conventional automated evaluation metrics and existing LLM-as-a-judge methods.