Evaluating Structured Output Robustness of Small Language Models for Open Attribute-Value Extraction from Clinical Notes

TL;DR

This study compares the robustness of small language models in generating structured clinical note data, finding JSON to be most parseable and highlighting factors affecting output quality for practical deployment.

Contribution

It provides a systematic evaluation of serialization formats and prompting strategies to improve structured output robustness in clinical NLP tasks.

Findings

JSON has highest parseability among formats

Larger models and targeted prompts improve robustness

Longer documents and specific note types reduce parseability

Abstract

We present a comparative analysis of the parseability of structured outputs generated by small language models for open attribute-value extraction from clinical notes. We evaluate three widely used serialization formats: JSON, YAML, and XML, and find that JSON consistently yields the highest parseability. Structural robustness improves with targeted prompting and larger models, but declines for longer documents and certain note types. Our error analysis identifies recurring format-specific failure patterns. These findings offer practical guidance for selecting serialization formats and designing prompts when deploying language models in privacy-sensitive clinical settings.