SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks

TL;DR

SpreadsheetArena introduces a platform for evaluating LLMs' ability to generate structured spreadsheet artifacts that meet diverse user constraints, highlighting the variability and challenges in aligning generated spreadsheets with domain-specific standards.

Contribution

We present SpreadsheetArena, a novel evaluation platform for assessing LLM-generated spreadsheets, emphasizing the complexity and variability of preferences across different use cases.

Findings

Preferences for spreadsheet features vary across use cases.

Even top-ranked models often do not align with domain-specific best practices.

Evaluation reveals significant variability in stylistic, structural, and functional features.

Abstract

Large language models (LLMs) are increasingly tasked with producing and manipulating structured artifacts. We consider the task of end-to-end spreadsheet generation, where language models are prompted to produce spreadsheet artifacts to satisfy users' explicit and implicit constraints, specified in natural language. We introduce SpreadsheetArena, a platform for evaluating models' performance on the task via blind pairwise evaluations of LLM-generated spreadsheet workbooks. As with other complex, open-ended tasks, relevant evaluation criteria can vary substantially across use cases and prompts, often in ways that are difficult to formalize. Compared to general chat or text generation settings, spreadsheet generation presents unique challenges and opportunities: the task output structure is well-defined and multi-dimensional, and there are often complex considerations around interactivity and layout. Among other findings, we observe that stylistic, structural, and functional features of preferred spreadsheets vary substantially across use cases, and expert evaluations of spreadsheets for finance prompts suggests that even highly ranked arena models do not reliably produce spreadsheets aligned with domain-specific best practices. Our hope is that our work prompts further study of end-to-end spreadsheet generation as a challenging and interesting category of complex, open-ended tasks for LLMs. Our live arena is hosted at https://spreadsheetarena.ai.