OPOR-Bench: Evaluating Large Language Models on Online Public Opinion Report Generation

TL;DR

This paper introduces OPOR-Bench, a new benchmark and evaluation framework for assessing large language models' ability to generate online public opinion reports during crises, addressing a significant research gap.

Contribution

It defines the OPOR-GEN task, creates the OPOR-BENCH dataset, and proposes OPOR-EVAL for systematic evaluation of report quality, facilitating future research in this area.

Findings

Frontier models perform well on the benchmark.

OPOR-EVAL correlates highly with human judgments.

The dataset covers 463 crisis events with diverse sources.

Abstract

Online Public Opinion Reports consolidate news and social media for timely crisis management by governments and enterprises. While large language models have made automated report generation technically feasible, systematic research in this specific area remains notably absent, particularly lacking formal task definitions and corresponding benchmarks. To bridge this gap, we define the Automated Online Public Opinion Report Generation (OPOR-GEN) task and construct OPOR-BENCH, an event-centric dataset covering 463 crisis events with their corresponding news articles, social media posts, and a reference summary. To evaluate report quality, we propose OPOR-EVAL, a novel agent-based framework that simulates human expert evaluation by analyzing generated reports in context. Experiments with frontier models demonstrate that our framework achieves high correlation with human judgments. Our comprehensive task definition, benchmark dataset, and evaluation framework provide a solid foundation for future research in this critical domain.