Schema Key Wording as an Instruction Channel in Structured Generation under Constrained Decoding

TL;DR

This paper investigates how schema key tokens serve as an implicit instruction channel in constrained decoding for structured generation, revealing their significant impact on model accuracy and behavior.

Contribution

It introduces a systematic study of schema keys as instruction channels, providing a theoretical analysis and empirical evidence of their influence across different models.

Findings

Schema key wording significantly affects accuracy in structured generation.

Qwen models benefit more from schema-level instructions, LLaMA models rely more on prompts.

Schema design is integral to instruction specification, not just output formatting.

Abstract

Constrained decoding is widely used to make large language models produce structured outputs that satisfy schemas such as JSON. Existing work mainly treats schemas as structural constraints, overlooking that schema-key tokens also enter the autoregressive context and may guide generation. To the best of our knowledge, we present the first systematic study of schema keys as an implicit instruction channel under constrained decoding. We formulate structured generation as a multi-channel instruction problem, where task signals can be placed in prompts, schema keys, or both. We further provide a projection-aware analysis: a CoT-style key helps only when its semantic gain exceeds the distortion induced by grammar-constrained projection, offering a theoretical explanation for model-dependent key effects. Experiments on mathematical reasoning benchmarks show that changing only schema-key wording can substantially affect accuracy while keeping the prompt, model, output structure, and decoding setup fixed. Qwen models tend to benefit more from schema-level instructions, whereas LLaMA models rely more on prompt-level guidance, and the two channels interact non-additively. Our findings show that schema design is not merely output formatting, but part of instruction specification in structured generation.