A note on the impossibility of conditional PAC-efficient reasoning in large language models

TL;DR

This paper proves that achieving conditional PAC-efficient reasoning in large language models is impossible in a distribution-free setting, showing that such models must rely heavily on expert models for most inputs.

Contribution

The paper establishes an impossibility result for conditional PAC-efficient reasoning, highlighting fundamental limitations in large language models' reasoning capabilities.

Findings

Conditional PAC efficiency cannot be achieved in non-atomic input spaces.

Any algorithm achieving conditional PAC efficiency must defer to expert models with high probability.

The result applies in the distribution-free setting, indicating fundamental limitations.

Abstract

We prove an impossibility result for conditional Probably Approximately Correct (PAC)-efficient reasoning in large language models. While recent work has established marginal PAC efficiency guarantees for composite models that switch between expensive expert models and cheaper fast models, we show that conditional (pointwise) guarantees are impossible in the distribution-free setting. Specifically, for non-atomic input spaces, any algorithm achieving conditional PAC efficiency must be trivial in the sense that it defers to the expert model with probability at least $1-\alpha$ for almost every input.