Metric assessment protocol in the context of answer fluctuation on MCQ tasks

TL;DR

This paper proposes a protocol for assessing metrics used in MCQ evaluations of language models, highlighting the impact of answer fluctuation and introducing a new metric called worst accuracy that correlates strongly with model stability.

Contribution

It introduces a metric assessment protocol linking evaluation metrics with answer fluctuation rates and proposes a novel metric, worst accuracy, for more reliable model evaluation.

Findings

Strong link between existing metrics and answer fluctuation.

Existing metrics correlate with answer changing even without prompt variations.

Worst accuracy shows the highest association with model stability.

Abstract

Using multiple-choice questions (MCQs) has become a standard for assessing LLM capabilities efficiently. A variety of metrics can be employed for this task. However, previous research has not conducted a thorough assessment of them. At the same time, MCQ evaluation suffers from answer fluctuation: models produce different results given slight changes in prompts. We suggest a metric assessment protocol in which evaluation methodologies are analyzed through their connection with fluctuation rates, as well as original performance. Our results show that there is a strong link between existing metrics and the answer changing, even when computed without any additional prompt variants. A novel metric, worst accuracy, demonstrates the highest association on the protocol.