Coherent Without Grounding, Grounded Without Success: Observability and Epistemic Failure

TL;DR

This paper challenges assumptions about understanding in LLMs by showing that observable success and explanation coherence can be dissociated, necessitating a new framework for evaluating artificial epistemic agents.

Contribution

It introduces the Bidirectional Coherence Paradox and the Epistemic Triangle model, highlighting the dissociation between success and explanation in LLMs across different observability domains.

Findings

LLMs can act successfully while misidentifying mechanisms in low-observability domains.

LLMs can generate explanations that are accurate but do not lead to effective intervention in high-observability domains.

Explanatory coherence alone does not indicate genuine understanding in LLMs.

Abstract

When an agent can articulate why something works, we typically take this as evidence of genuine understanding. This presupposes that effective action and correct explanation covary, and that coherent explanation reliably signals both. I argue that this assumption fails for contemporary Large Language Models (LLMs). I introduce what I call the Bidirectional Coherence Paradox: competence and grounding not only dissociate but invert across epistemic conditions. In low-observability domains, LLMs often act successfully while misidentifying the mechanisms that produce their success. In high-observability domains, they frequently generate explanations that accurately track observable causal structure yet fail to translate those diagnoses into effective intervention. In both cases, explanatory coherence remains intact, obscuring the underlying dissociation. Drawing on experiments in compiler optimization and hyperparameter tuning, I develop the Epistemic Triangle, a model of how priors, signals, and domain knowledge interact under varying observability. The results suggest that neither behavioral success nor explanatory accuracy alone suffices for attributing understanding. I argue that evaluating artificial epistemic agents requires a tripartite framework -- coherence, grounding, and a proper basing relation linking explanation to action. The systematic separation of knowing-that and knowing-how in LLMs thus challenges assumptions inherited from both epistemology and current AI evaluation practice.