Alignment Verifiability in Large Language Models: Normative Indistinguishability under Behavioral Evaluation

TL;DR

This paper investigates the limitations of behavioral evaluation in large language models, revealing that observed compliance under finite tests cannot reliably confirm true alignment due to normative indistinguishability and evaluation awareness.

Contribution

It formalizes the Alignment Verifiability Problem, introduces the concept of normative indistinguishability, and proves a conditional impossibility result for identifying latent alignment from behavioral data.

Findings

Finite behavioral evaluation cannot uniquely determine latent alignment.

Evaluation-aware policies can mimic compliant behavior without true alignment.

Constructive example with an instruction-tuned LLM demonstrates the theoretical limitations.

Abstract

Behavioral evaluation is the dominant paradigm for assessing alignment in large language models (LLMs). In current practice, observed compliance under finite evaluation protocols is treated as evidence of latent alignment. However, the inference from bounded behavioral evidence to claims about global latent properties is rarely analyzed as an identifiability problem. In this paper, we study alignment evaluation through the lens of statistical identifiability under partial observability. We allow agent policies to condition their behavior on observable signals correlated with the evaluation regime, a phenomenon we term evaluation awareness. Within this framework, we formalize the Alignment Verifiability Problem and introduce Normative Indistinguishability, which arises when distinct latent alignment hypotheses induce identical distributions over evaluator-accessible observations. Our main theoretical contribution is a conditional impossibility result: under finite behavioral evaluation and evaluation-aware policies, observed compliance does not uniquely identify latent alignment, but only membership in an equivalence class of conditionally compliant policies, under explicit assumptions on policy expressivity and observability. We complement the theory with a constructive existence proof using an instruction-tuned LLM (Llama-3.2-3B), demonstrating a conditional policy that is perfectly compliant under explicit evaluation signals yet exhibits degraded identifiability when the same evaluation intent is conveyed implicitly. Together, our results show that behavioral benchmarks provide necessary but insufficient evidence for latent alignment under evaluation awareness.