Factual Inconsistency in Data-to-Text Generation Scales Exponentially with LLM Size: A Statistical Validation

TL;DR

This study rigorously investigates how factual inconsistency in data-to-text generation scales with large language model size, revealing that it follows an exponential pattern rather than the commonly assumed power law.

Contribution

The paper introduces a statistical validation framework and demonstrates that factual inconsistency scales exponentially with LLM size, challenging previous assumptions of power law scaling.

Findings

Factual inconsistency increases exponentially with LLM size.

Empirical analysis across multiple datasets supports exponential scaling.

Contradicts the common belief of power law scaling in LLMs.

Abstract

Monitoring factual inconsistency is essential for ensuring trustworthiness in data-to-text generation (D2T). While large language models (LLMs) have demonstrated exceptional performance across various D2T tasks, previous studies on scaling laws have primarily focused on generalization error through power law scaling to LLM size (i.e., the number of model parameters). However, no research has examined the impact of LLM size on factual inconsistency in D2T. In this paper, we investigate how factual inconsistency in D2T scales with LLM size by exploring two scaling laws: power law and exponential scaling. To rigorously evaluate and compare these scaling laws, we employ a statistical validation framework consisting of three key stages: predictive performance estimation, goodness-of-fit assessment, and comparative analysis. For a comprehensive empirical study, we analyze three popular LLM families across five D2T datasets, measuring factual inconsistency inversely using four state-of-the-art consistency metrics. Our findings, based on exhaustive empirical results and validated through our framework, reveal that, contrary to the widely assumed power law scaling, factual inconsistency in D2T follows an exponential scaling with LLM size.