VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents

TL;DR

VAREX is a comprehensive benchmark for evaluating multimodal foundation models on structured data extraction from government forms, emphasizing input modality effects and model scalability.

Contribution

It introduces a new benchmark with synthetic, multi-modal documents, enabling systematic analysis of input formats and model performance, especially for small-scale models.

Findings

Models under 4B parameters struggle with schema compliance, reducing scores by 45-65 pp.

Fine-tuning at 2B parameters significantly improves extraction accuracy (+81 pp).

Layout-preserving text enhances accuracy more than visual cues, with gains of 3-18 pp.

Abstract

We introduce VAREX (VARied-schema EXtraction), a benchmark for evaluating multimodal foundation models on structured data extraction from government forms. VAREX employs a Reverse Annotation pipeline that programmatically fills PDF templates with synthetic values, producing deterministic ground truth validated through three-phase quality assurance. The benchmark comprises 1,777 documents with 1,771 unique schemas across three structural categories, each provided in four input modalities: plain text, layout-preserving text (whitespace-aligned to approximate column positions), document image, or both text and image combined. Unlike existing benchmarks that evaluate from a single input representation, VAREX provides four controlled modalities per document, enabling systematic ablation of how input format affects extraction accuracy -- a capability absent from prior benchmarks. We evaluate 20 models from frontier proprietary models to small open models, with particular attention to models <=4B parameters suitable for cost-sensitive and latency-constrained deployment. Results reveal that (1) below 4B parameters, structured output compliance -- not extraction capability -- is a dominant bottleneck; in particular, schema echo (models producing schema-conforming structure instead of extracted values) depresses scores by 45-65 pp (percentage points) in affected models; (2) extraction-specific fine-tuning at 2B yields +81 pp gains, demonstrating that the instruction-following deficit is addressable without scale; (3) layout-preserving text provides the largest accuracy gain (+3-18 pp), exceeding pixel-level visual cues; and (4) the benchmark most effectively discriminates models in the 60-95% accuracy band. Dataset and evaluation code are publicly available.