Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks

TL;DR

This paper critically examines the effectiveness of human pairwise preferences in meta-evaluating long-form QA benchmarks, highlighting limitations and proposing guidelines for improved evaluation practices in scientific report generation systems.

Contribution

It provides a comprehensive case study on meta-evaluation methods for scientific QA benchmarks, revealing the strengths and weaknesses of pairwise preferences and offering practical guidelines for future evaluations.

Findings

Pairwise preferences are effective at system-level evaluation.

Explicit metric annotations are crucial for metric-level assessment.

Subjectivity remains a key challenge in human evaluations.

Abstract

Recent advances have made long-form report-generating systems widely available. This has prompted evaluation frameworks that use LLM-as-judge protocols and claim verification, along with meta-evaluation frameworks that seek to validate these methods. Many of the meta-evaluations estimate an evaluation quality's by comparing its assessments against human pairwise preferences. Prior work, however, suggests that human pairwise preference may be overly simplistic and can fail to capture nuances of expert expectations. We conduct a case study in meta-evaluation for long-form QA benchmarks using ScholarQA-CS2, a benchmark designed for assessing retrieval-augmented deep-research QA in the scientific domain. We comprehensively validate the benchmark through human pairwise preference judgments, then critically examine the strengths, weaknesses, and confounders of this approach. We show that pairwise preference rankings are best suited for system-level evaluation, while explicit metric-wise annotations and expert annotators are critical for reliable metric-level assessment, with subjectivity remaining a key challenge. Based on our findings, we offer practical guidelines for designing future meta-evaluations that better align evaluation methods, annotator expertise, and reporting practices. By surfacing these methodological challenges, we aim to advance evaluation standards for deep-research systems.