Generating Tables from the Parametric Knowledge of Language Models

TL;DR

This paper investigates the ability of large language models to generate accurate and structured tables from their knowledge, introduces a new benchmark dataset, and analyzes factors affecting table generation quality.

Contribution

It presents a comprehensive evaluation of LLMs for table generation, introduces the WikiTabGen benchmark, and analyzes how table properties impact generation accuracy.

Findings

GPT-4 achieves up to 19.6% accuracy in table generation.

Table size and numerical content influence generation performance.

Table popularity affects the accuracy of generated tables.

Abstract

We explore generating factual and accurate tables from the parametric knowledge of large language models (LLMs). While LLMs have demonstrated impressive capabilities in recreating knowledge bases and generating free-form text, we focus on generating structured tabular data, which is crucial in domains like finance and healthcare. We examine the table generation abilities of four state-of-the-art LLMs: GPT-3.5, GPT-4, Llama2-13B, and Llama2-70B, using three prompting methods for table generation: (a) full-table, (b) row-by-row; (c) cell-by-cell. For evaluation, we introduce a novel benchmark, WikiTabGen which contains 100 curated Wikipedia tables. Tables are further processed to ensure their factual correctness and manually annotated with short natural language descriptions. Our findings reveal that table generation remains a challenge, with GPT-4 reaching the highest accuracy at 19.6%. Our detailed analysis sheds light on how various table properties, such as size, table popularity, and numerical content, influence generation performance. This work highlights the unique challenges in LLM-based table generation and provides a solid evaluation framework for future research. Our code, prompts and data are all publicly available: https://github.com/analysis-bots/WikiTabGen