Lost in Space: Finding the Right Tokens for Structured Output

TL;DR

This paper investigates how different structured output formats affect the performance of language models in NLP tasks, finding that formats aligned with conventions and including leading whitespace improve accuracy and robustness.

Contribution

It systematically compares the impact of various output formats on model performance and provides best practices for using structured outputs in zero-shot classification.

Findings

Formats respecting conventions improve accuracy by 5-10%.

Including leading whitespace enhances model performance, especially for smaller models.

Structured output formats influence downstream task effectiveness.

Abstract

General-purpose language models are trained to produce varied natural language outputs, but for some tasks, like annotation or classification, we need more specific output formats. LLM systems increasingly support structured output, which enforces formats by sampling tokens according to a grammar -- but also unpredictably reduces downstream performance. Are there systematic differences between grammars that appear semantically (and often visually) similar to humans? To answer this, we test four popular model families with five varying output formats on four common NLP benchmarks. We find all models perform most accurately when guided to use formats respecting convention, such as letters for multiple choice and real numbers for numerical prediction. Performance also improves by 5%-10% when guiding models to return tokens incorporating leading whitespace, with smaller models benefiting the most. We find leading whitespace helps models avoid structural deficiencies in subword token representations. We finally present best practices for researchers using language models as zero-shot classifiers with structured output.