Beyond Hallucinations: A Composite Score for Measuring Reliability in Open-Source Large Language Models

TL;DR

This paper introduces the Composite Reliability Score (CRS), a unified metric to evaluate the reliability of open-source Large Language Models across multiple dimensions like calibration, robustness, and uncertainty, addressing fragmented existing evaluations.

Contribution

The paper proposes the CRS framework that combines various reliability aspects into a single interpretable score, providing comprehensive evaluation of open-source LLMs.

Findings

CRS provides stable model rankings across datasets.

It uncovers hidden failure modes missed by single metrics.

Dependable models balance accuracy, robustness, and calibrated uncertainty.

Abstract

Large Language Models (LLMs) like LLaMA, Mistral, and Gemma are increasingly used in decision-critical domains such as healthcare, law, and finance, yet their reliability remains uncertain. They often make overconfident errors, degrade under input shifts, and lack clear uncertainty estimates. Existing evaluations are fragmented, addressing only isolated aspects. We introduce the Composite Reliability Score (CRS), a unified framework that integrates calibration, robustness, and uncertainty quantification into a single interpretable metric. Through experiments on ten leading open-source LLMs across five QA datasets, we assess performance under baselines, perturbations, and calibration methods. CRS delivers stable model rankings, uncovers hidden failure modes missed by single metrics, and highlights that the most dependable systems balance accuracy, robustness, and calibrated uncertainty.