LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiency

TL;DR

This paper models LLM evaluation data as a low-rank tensor completion problem under pairwise comparisons, deriving efficient estimators and uncertainty quantification methods for structured, noisy, and non-uniform data.

Contribution

It introduces a semiparametric inference framework with a score-whitening method for stable, optimal uncertainty quantification in low-rank tensor models from pairwise data.

Findings

Derived the semiparametric efficiency bound for LLM evaluation tensors.

Constructed a one-step debiased estimator with asymptotic normality.

Introduced a score-whitening technique to stabilize inference in anisotropic models.

Abstract

Large language model (LLM) evaluation platforms increasingly rely on pairwise human judgments. These data are noisy, sparse, and non-uniform, yet leaderboards are reported with limited uncertainty quantification. We study this as semiparametric inference for a low-rank latent score tensor observed through pairwise comparisons under Bradley-Terry-Luce-type models. This places LLM evaluation in a new tensor completion setting with structured observations, non-uniform sampling, and pairwise contrasts. Our target is a smooth functional $\psi(T^\star)$, including linear estimands such as ability gaps and nonlinear ones such as win probabilities. We derive the information operator on the low-rank tangent space, the efficient influence function, and the semiparametric efficiency bound, then construct a one-step debiased estimator with asymptotic normality. A central challenge is that the information operator is anisotropic and does not commute with the tangent-space projection, creating a bottleneck absent from isotropic models. We introduce a score-whitening method that equalizes local Fisher information and restores stable inference at the optimal sample-complexity scale. Our results provide a principled framework for uncertainty quantification in LLM evaluation and more broadly for inference on low-rank structures from pairwise data.