Why the Valuable Capabilities of LLMs Are Precisely the Unexplainable Ones

TL;DR

This paper argues that the most valuable capabilities of large language models are inherently unexplainable and cannot be fully captured by human-readable rules, highlighting a fundamental limit of interpretability.

Contribution

It presents a counterintuitive thesis supported by expert system theory and philosophical concepts, emphasizing the intrinsic unexplainability of LLM capabilities.

Findings

LLMs possess capabilities beyond rule-based systems.

Expert systems are demonstrably weaker than LLMs.

Unexplainable capabilities are central to LLM value.

Abstract

This paper proposes and argues for a counterintuitive thesis: the truly valuable capabilities of large language models (LLMs) reside precisely in the part that cannot be fully captured by human-readable discrete rules. The core argument is a proof by contradiction via expert system equivalence: if the full capabilities of an LLM could be described by a complete set of human-readable rules, then that rule set would be functionally equivalent to an expert system; but expert systems have been historically and empirically demonstrated to be strictly weaker than LLMs; therefore, a contradiction arises -- the capabilities of LLMs that exceed those of expert systems are exactly the capabilities that cannot be rule-encoded. This thesis is further supported by the Chinese philosophical concept of Wu (sudden insight through practice), the historical failure of expert systems, and a structural mismatch between human cognitive tools and complex systems. The paper discusses implications for interpretability research, AI safety, and scientific epistemology.