PromptPort: A Reliability Layer for Cross-Model Structured Extraction

TL;DR

PromptPort enhances the reliability of structured extraction from LLMs by addressing format collapse issues through canonicalization, verification, and abstention, significantly improving operational robustness without altering base models.

Contribution

The paper introduces PromptPort, a novel reliability layer that mitigates format collapse and cross-model variability in structured extraction tasks.

Findings

Severe format collapse observed across models (ROS 0.116 vs CSS 0.246).

PromptPort recovers 6-8 F1 points through canonicalization and verification.

Approaches oracle-level performance (0.890 vs 0.896 F1) without modifying base models.

Abstract

Structured extraction with LLMs fails in production not because models lack understanding, but because output formatting is unreliable across models and prompts. A prompt that returns clean JSON on GPT-4 may produce fenced, prose-wrapped, or malformed output on Llama, causing strict parsers to reject otherwise correct extractions. We formalize this as format collapse and introduce a dual-metric evaluation framework: ROS (strict parsing, measuring operational reliability) and CSS (post-canonicalization, measuring semantic capability). On a 37,346-example camera metadata benchmark across six model families, we find severe format collapse (for example, Gemma-2B: ROS 0.116 versus CSS 0.246) and large cross-model portability gaps (0.4 to 0.6 F1). We then present PromptPort, a reliability layer combining deterministic canonicalization with a lightweight verifier (DistilBERT) and a safe-override policy. PromptPort recovers format failures (plus 6 to 8 F1), adds verifier-driven semantic selection (plus 14 to 16 F1 beyond canonicalization), and approaches per-field oracle performance (0.890 versus 0.896 in zero-shot) without modifying base models. The method generalizes to held-out model families and provides explicit abstention when uncertain, enabling reliable structured extraction in production deployments.