Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements

TL;DR

This paper introduces PrecisionDiff, a framework for systematically detecting subtle behavioral disagreements in large language models caused by different numerical precisions, which are often overlooked by standard evaluation methods.

Contribution

The paper presents a novel automated differential testing framework that uncovers precision-induced behavioral disagreements in LLMs, improving evaluation and robustness.

Findings

Behavioral disagreements are widespread across open-source LLMs and precision settings.

PrecisionDiff significantly outperforms traditional testing methods in detecting these issues.

Precision-induced divergences can lead to harmful responses in certain cases.

Abstract

Large language models (LLMs) are increasingly deployed under diverse numerical precision configurations, including standard floating-point formats (e.g., bfloat16 and float16) and quantized integer formats (e.g., int16 and int8), to meet efficiency and resource constraints. However, minor inconsistencies between LLMs of different precisions are difficult to detect and are often overlooked by existing evaluation methods. In this paper, we present PrecisionDiff, an automated differential testing framework for systematically detecting precision-induced behavioral disagreements in LLMs. PrecisionDiff generates precision-sensitive test inputs and performs cross-precision comparative analysis to uncover subtle divergences that remain hidden under conventional testing strategies. To demonstrate its practical significance, we instantiate PrecisionDiff on the alignment verification task, where precision-induced disagreements manifest as jailbreak divergence-inputs that are rejected under one precision may produce harmful responses under another. Experimental results show that such behavioral disagreements are widespread across multiple open-source aligned LLMs and precision settings, and that PrecisionDiff significantly outperforms vanilla testing methods in detecting these issues. Our work enables automated precision-sensitive test generation, facilitating effective pre-deployment evaluation and improving precision robustness during training.