DataTales: A Benchmark for Real-World Intelligent Data Narration

TL;DR

DataTales is a new benchmark that evaluates language models on their ability to generate clear, analytical narratives from complex financial data, addressing a gap in existing evaluation tools.

Contribution

It introduces a large, real-world dataset and benchmark for assessing data narration capabilities of language models in financial contexts.

Findings

Language models struggle with precision and analytical depth in data narration.

The benchmark reveals significant challenges faced by current models.

Provides a foundation for future model improvements and evaluation methods.

Abstract

We introduce DataTales, a novel benchmark designed to assess the proficiency of language models in data narration, a task crucial for transforming complex tabular data into accessible narratives. Existing benchmarks often fall short in capturing the requisite analytical complexity for practical applications. DataTales addresses this gap by offering 4.9k financial reports paired with corresponding market data, showcasing the demand for models to create clear narratives and analyze large datasets while understanding specialized terminology in the field. Our findings highlights the significant challenge that language models face in achieving the necessary precision and analytical depth for proficient data narration, suggesting promising avenues for future model development and evaluation methodologies.