PodBench: A Comprehensive Benchmark for Instruction-Aware Audio-Oriented Podcast Script Generation

TL;DR

PodBench is a new comprehensive benchmark designed to evaluate instruction-aware, audio-oriented podcast script generation, addressing the lack of systematic evaluation resources for complex multi-speaker dialogue synthesis.

Contribution

We introduce PodBench, a large-scale benchmark with a novel evaluation framework and extensive experiments, highlighting strengths and limitations of current models in long-form, multi-speaker podcast script generation.

Findings

Proprietary models outperform open-source models overall.

Open-source models with explicit reasoning show better robustness.

High instruction following does not always ensure high content quality.

Abstract

Podcast script generation requires LLMs to synthesize structured, context-grounded dialogue from diverse inputs, yet systematic evaluation resources for this task remain limited. To bridge this gap, we introduce PodBench, a benchmark comprising 800 samples with inputs up to 21K tokens and complex multi-speaker instructions. We propose a multifaceted evaluation framework that integrates quantitative constraints with LLM-based quality assessment. Extensive experiments reveal that while proprietary models generally excel, open-source models equipped with explicit reasoning demonstrate superior robustness in handling long contexts and multi-speaker coordination compared to standard baselines. However, our analysis uncovers a persistent divergence where high instruction following does not guarantee high content substance. PodBench offers a reproducible testbed to address these challenges in long-form, audio-centric generation.