Elicitation Matters: How Prompts and Query Protocols Shape LLM Surrogates under Sparse Observations

TL;DR

This paper investigates how prompt design and query protocols influence the uncertainty and predictions of LLMs used as surrogates in low-data optimization, highlighting the importance of elicitation strategies.

Contribution

It introduces an uncertainty-alignment criterion and demonstrates that prompt structure and query methods significantly affect LLM surrogate behavior and optimization outcomes.

Findings

Structural prompts serve as effective priors.

Different query protocols lead to distinct belief formations.

Sequential evidence causes non-monotonic confidence updates.

Abstract

Large language models are increasingly used as surrogate models for low-data optimization, but their optimizer-facing prediction and its uncertainty remain poorly understood. We study the surrogate belief elicited from an LLM under sparse observations, showing that it depends strongly on prompt text and query protocol. We introduce an uncertainty-alignment criterion that measures whether model uncertainty tracks residual ambiguity among sample-consistent functions. Across controlled inference tasks and Bayesian optimization studies, we find that structural prompts act as effective priors, POINTWISE and JOINT querying induce different beliefs, and sequential evidence leads to non-monotonic, order-sensitive confidence updates. These effects change downstream acquisition decisions and regret, showing that elicitation protocol is part of the LLM surrogate specification, not a formatting detail.