Moneyball with LLMs: Analyzing Tabular Summarization in Sports Narratives

TL;DR

This paper introduces SPORTABSET, a benchmark for evaluating long-context tabular summarization in sports narratives, revealing challenges in model memory, robustness, and multi-entity tracking for LLMs.

Contribution

The paper presents SPORTABSET, a diagnostic benchmark for assessing long-context tabular summarization, and systematically analyzes the limitations of current LLM strategies in this domain.

Findings

Decomposition improves accuracy mainly by reducing multi-entity interference.

Models are highly sensitive to surface cues, leading to hallucinations and omissions.

Memory for multiple entities is a key bottleneck in long-context summarization.

Abstract

Large language model (LLM) approaches to tabular summarization rely on extensive prompt engineering, decomposition pipelines, or entity-level intermediate representations to achieve strong performance. While effective, these strategies are computationally expensive and offer limited insight into how well models maintain state over long, evolving narratives. We introduce SPORTABSET, a diagnostic benchmark for long-context tabular summarization across two complementary sports domains that require tracking multiple entities and aggregating statistics under domain-specific rules. Using SporTabSet, we systematically evaluate decomposition-based strategies across several long context LLMs. Results show that although decomposition substantially improves accuracy and numerical fidelity, gains stem mainly from dissecting multi-entity interference rather than improved local arithmetic. Robustness experiments further reveal high sensitivity to surface-level cues with structured failures, including hallucination, omission, and role confusion. Together, these findings identify consistent multientity memory as a key bottleneck in long context table generation, motivating diagnostic evaluation as a prerequisite for scalable, efficient and reliable tabular summarization models.