DynamicBench: Evaluating Real-Time Report Generation in Large Language Models

TL;DR

DynamicBench is a new benchmark for evaluating large language models' ability to generate real-time reports using up-to-date information retrieval and processing, addressing limitations of static evaluation methods.

Contribution

We introduce DynamicBench, a novel benchmark and a report generation system that assess LLMs' proficiency in handling dynamic, real-time data in specialized domains.

Findings

Our method surpasses GPT4o by 7.0% in document-free scenarios.

Achieves 5.8% higher performance in document-assisted scenarios.

Demonstrates state-of-the-art results in dynamic information synthesis.

Abstract

Traditional benchmarks for large language models (LLMs) typically rely on static evaluations through storytelling or opinion expression, which fail to capture the dynamic requirements of real-time information processing in contemporary applications. To address this limitation, we present DynamicBench, a benchmark designed to evaluate the proficiency of LLMs in storing and processing up-to-the-minute data. DynamicBench utilizes a dual-path retrieval pipeline, integrating web searches with local report databases. It necessitates domain-specific knowledge, ensuring accurate responses report generation within specialized fields. By evaluating models in scenarios that either provide or withhold external documents, DynamicBench effectively measures their capability to independently process recent information or leverage contextual enhancements. Additionally, we introduce an advanced report generation system adept at managing dynamic information synthesis. Our experimental results confirm the efficacy of our approach, with our method achieving state-of-the-art performance, surpassing GPT4o in document-free and document-assisted scenarios by 7.0% and 5.8%, respectively. The code and data will be made publicly available.