PertEval: Unveiling Real Knowledge Capacity of LLMs with Knowledge-Invariant Perturbations

TL;DR

PertEval is a toolkit that uses knowledge-invariant perturbations to more accurately assess the true knowledge capacity of large language models, revealing overestimations in current benchmarks and exposing models' weaknesses.

Contribution

We introduce PertEval, a novel probing toolkit employing human-like restatement perturbations to evaluate LLMs' genuine knowledge, reducing bias from test scenario limitations and data contamination.

Findings

LLMs' performance is significantly overestimated on raw benchmarks.

PertEval reveals models' uncertainty and rote memorization tendencies.

Response consistency analysis uncovers weaknesses in LLMs' knowledge mastery.

Abstract

Expert-designed close-ended benchmarks are indispensable in assessing the knowledge capacity of large language models (LLMs). Despite their widespread use, concerns have mounted regarding their reliability due to limited test scenarios and an unavoidable risk of data contamination. To rectify this, we present PertEval, a toolkit devised for in-depth probing of LLMs' knowledge capacity through \textbf{knowledge-invariant perturbations}. These perturbations employ human-like restatement techniques to generate on-the-fly test samples from static benchmarks, meticulously retaining knowledge-critical content while altering irrelevant details. Our toolkit further includes a suite of \textbf{response consistency analyses} that compare performance on raw vs. perturbed test sets to precisely assess LLMs' genuine knowledge capacity. Six representative LLMs are re-evaluated using PertEval. Results reveal significantly inflated performance of the LLMs on raw benchmarks, including an absolute 25.8% overestimation for GPT-4. Additionally, through a nuanced response pattern analysis, we discover that PertEval retains LLMs' uncertainty to specious knowledge, and reveals their potential rote memorization to correct options which leads to overestimated performance. We also find that the detailed response consistency analyses by PertEval could illuminate various weaknesses in existing LLMs' knowledge mastery and guide the development of refinement. Our findings provide insights for advancing more robust and genuinely knowledgeable LLMs. Our code is available at \url{https://github.com/aigc-apps/PertEval}.