When LLMs get significantly worse: A statistical approach to detect model degradations

TL;DR

This paper introduces a statistical hypothesis testing framework using McNemar's test to reliably detect model degradations in large language models, ensuring accuracy changes are genuine rather than noise.

Contribution

It proposes a novel, statistically sound method for identifying model degradation by comparing sample-level scores, with aggregation techniques for multiple benchmarks, implemented on open-source tools.

Findings

The method correctly flags degraded models while ignoring harmless optimizations.

It can detect accuracy degradations as small as 0.3%.

The approach guarantees a controlled false positive rate.

Abstract

Minimizing the inference cost and latency of foundation models has become a crucial area of research. Optimization approaches include theoretically lossless methods and others without accuracy guarantees like quantization. In all of these cases it is crucial to ensure that the model quality has not degraded. However, even at temperature zero, model generations are not necessarily robust even to theoretically lossless model optimizations due to numerical errors. We thus require statistical tools to decide whether a finite-sample accuracy deviation is an evidence of a model's degradation or whether it can be attributed to (harmless) noise in the evaluation. We propose a statistically sound hypothesis testing framework based on McNemar's test allowing to efficiently detect model degradations, while guaranteeing a controlled rate of false positives. The crucial insight is that we have to confront the model scores on each sample, rather than aggregated on the task level. Furthermore, we propose three approaches to aggregate accuracy estimates across multiple benchmarks into a single decision. We provide an implementation on top of the largely adopted open source LM Evaluation Harness and provide a case study illustrating that the method correctly flags degraded models, while not flagging model optimizations that are provably lossless. We find that with our tests even empirical accuracy degradations of 0.3% can be confidently attributed to actual degradations rather than noise.