Eye of Judgement: Dissecting the Evaluation of Russian-speaking LLMs with POLLUX

TL;DR

This paper introduces POLLUX, an open-source benchmark with a novel interpretability-focused evaluation methodology for Russian LLMs, including a detailed taxonomy, scoring protocol, and LLM-based evaluators.

Contribution

The paper presents a new evaluation framework for Russian LLMs that emphasizes interpretability and transparency, along with a comprehensive benchmark dataset and LLM-based evaluators.

Findings

POLLUX enables transparent, criteria-driven evaluation of LLMs.

The benchmark covers 35 diverse task types with 2,100 prompts.

LLM-based evaluators provide nuanced assessments comparable to human judgments.

Abstract

We introduce POLLUX, a comprehensive open-source benchmark designed to evaluate the generative capabilities of large language models (LLMs) in Russian. Our main contribution is a novel evaluation methodology that enhances the interpretability of LLM assessment. For each task type, we define a set of detailed criteria and develop a scoring protocol where models evaluate responses and provide justifications for their ratings. This enables transparent, criteria-driven evaluation beyond traditional resource-consuming, side-by-side human comparisons. POLLUX includes a detailed, fine-grained taxonomy of 35 task types covering diverse generative domains such as code generation, creative writing, and practical assistant use cases, totaling 2,100 manually crafted and professionally authored prompts. Each task is categorized by difficulty (easy/medium/hard), with experts constructing the dataset entirely from scratch. We also release a family of LLM-as-a-Judge (7B and 32B) evaluators trained for nuanced assessment of generative outputs. This approach provides scalable, interpretable evaluation and annotation tools for model development, effectively replacing costly and less precise human judgments.