Evaluating Structured Decoding for Text-to-Table Generation: Evidence from Three Datasets

TL;DR

This paper systematically evaluates structured decoding versus standard prompting for text-to-table generation across three datasets, showing that structured decoding improves validity and alignment in resource-limited large language models.

Contribution

It provides the first comprehensive comparison of schema-guided structured decoding against unstructured prompting in text-to-table generation across multiple benchmarks.

Findings

Structured decoding improves table validity and alignment.

Performance varies with dataset and task complexity.

Model size influences decoding effectiveness.

Abstract

We present a comprehensive evaluation of structured decoding for text-to-table generation with large language models (LLMs). While previous work has primarily focused on unconstrained generation of tables, the impact of enforcing structural constraints during generation remains underexplored. We systematically compare schema-guided (structured) decoding to standard one-shot prompting across three diverse benchmarks - E2E, Rotowire, and Livesum - using open-source LLMs of up to 32B parameters, assessing the performance of table generation approaches in resource-constrained settings. Our experiments cover a wide range of evaluation metrics at cell, row, and table levels. Results demonstrate that structured decoding significantly enhances the validity and alignment of generated tables, particularly in scenarios demanding precise numerical alignment (Rotowire), but may degrade performance in contexts involving densely packed textual information (E2E) or extensive aggregation over lengthy texts (Livesum). We further analyze the suitability of different evaluation metrics and discuss the influence of model size.