A CAP-like Trilemma for Large Language Models: Correctness, Non-bias, and Utility under Semantic Underdetermination

TL;DR

This paper introduces a CAP-like trilemma for Large Language Models, showing they cannot simultaneously ensure correctness, non-bias, and utility under semantic underdetermination, highlighting fundamental limitations in their responses.

Contribution

It formulates a novel trilemma for LLMs inspired by the CAP theorem, connecting semantic underdetermination to core response trade-offs.

Findings

LLMs face a fundamental trade-off between correctness, non-bias, and utility.

Underdetermined prompts lead to biases or reduced utility in LLM responses.

Certain LLM failures stem from the inherent structure of underdetermined decision requests.

Abstract

The CAP theorem states that a distributed system cannot simultaneously guarantee consistency, availability, and partition tolerance under network partition. Inspired by this result, this paper formulates a CAP-like conjecture for Large Language Models (LLMs). The proposed trilemma states that, under semantic underdetermination, an LLM cannot always simultaneously guarantee strong correctness, strict non-bias, and high utility. A prompt is semantically underdetermined when the given premises do not determine a unique answer. In such cases, a useful and decisive response requires the model to introduce a selection criterion, preference, prior, or value ordering. If this criterion is not supplied by the user or justified by the available premises, the response becomes biased in a broad selection-theoretic sense. Conversely, if the model avoids unsupported preferences, it may preserve correctness and non-bias but may reduce utility through refusal, hedging, or clarification. The paper formalizes this correctness--non-bias--utility trilemma, develops examples, and argues that certain LLM failures arise not merely from model limitations but from the structure of underdetermined decision requests.