Checklist Engineering Empowers Multilingual LLM Judges
Mohammad Ghiasvand Mohammadkhani, Hamid Beigy

TL;DR
This paper introduces CE-Judge, a training-free, checklist-based framework for multilingual evaluation using open-source LLMs, achieving competitive performance across multiple languages and benchmarks without extensive training.
Contribution
The paper presents a novel, training-free checklist engineering approach for multilingual LLM evaluation, reducing reliance on proprietary models and extensive data.
Findings
CE-Judge outperforms baseline methods in multilingual evaluation.
CE-Judge performs comparably to GPT-4o on benchmark datasets.
The framework is effective under both pointwise and pairwise settings.
Abstract
Automated text evaluation has long been a central issue in Natural Language Processing (NLP). Recently, the field has shifted toward using Large Language Models (LLMs) as evaluators-a trend known as the LLM-as-a-Judge paradigm. While promising and easily adaptable across tasks, this approach has seen limited exploration in multilingual contexts. Existing multilingual studies often rely on proprietary models or require extensive training data for fine-tuning, raising concerns about cost, time, and efficiency. In this paper, we propose Checklist Engineering based LLM-as-a-Judge (CE-Judge), a training-free framework that uses checklist intuition for multilingual evaluation with an open-source model. Experiments across multiple languages and three benchmark datasets, under both pointwise and pairwise settings, show that our method generally surpasses the baselines and performs on par with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Computational and Text Analysis Methods · Natural Language Processing Techniques
