Predicting Question-Answering Performance of Large Language Models through Semantic Consistency

TL;DR

This paper introduces a new benchmark dataset and a framework that uses semantic consistency and other metrics to predict the factual question-answering accuracy of large language models without needing reference answers.

Contribution

It presents a manually created high-quality paraphrase dataset for factual questions and a novel framework combining semantic consistency with existing metrics for performance prediction.

Findings

Framework significantly outperforms baselines

High correlation between semantic consistency and QA accuracy

Effective for multiple large language models

Abstract

Semantic consistency of a language model is broadly defined as the model's ability to produce semantically-equivalent outputs, given semantically-equivalent inputs. We address the task of assessing question-answering (QA) semantic consistency of contemporary large language models (LLMs) by manually creating a benchmark dataset with high-quality paraphrases for factual questions, and release the dataset to the community. We further combine the semantic consistency metric with additional measurements suggested in prior work as correlating with LLM QA accuracy, for building and evaluating a framework for factual QA reference-less performance prediction -- predicting the likelihood of a language model to accurately answer a question. Evaluating the framework on five contemporary LLMs, we demonstrate encouraging, significantly outperforming baselines, results.