PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relations

TL;DR

This paper presents a novel confidence scoring method for zero-shot LLMs in textual classification, using metamorphic relations to assess response consistency, leading to improved accuracy in single and ensemble settings.

Contribution

Introduces Perceived Confidence Score (PCS), a new technique leveraging metamorphic relations to evaluate and enhance zero-shot LLM classification performance.

Findings

PCS improves zero-shot LLM classification accuracy by 9.3%.

Using PCS in ensemble setups boosts performance by 5.8%.

Metamorphic testing effectively assesses LLM response consistency.

Abstract

Zero-shot LLMs are now also used for textual classification tasks, e.g., sentiment and bias detection in a sentence or article. However, their performance can be suboptimal in such data annotation tasks. We introduce a novel technique that evaluates an LLM's confidence for classifying a textual input by leveraging Metamorphic Relations (MRs). The MRs generate semantically equivalent yet textually divergent versions of the input. Following the principles of Metamorphic Testing (MT), the mutated versions are expected to have annotation labels similar to the input. By analyzing the consistency of an LLM's responses across these variations, we compute a perceived confidence score (PCS) based on the frequency of the predicted labels. PCS can be used for both single and multiple LLM settings (e.g., when multiple LLMs are vetted in a majority-voting setup). Empirical evaluation shows that our PCS-based approach improves the performance of zero-shot LLMs by 9.3% in textual classification tasks. When multiple LLMs are used in a majority-voting setup, we obtain a performance boost of 5.8% with PCS.