TL;DR
The paper introduces DoGMaTiQ, an automated pipeline for generating question-answer nuggets to evaluate reports, especially in multilingual and cross-lingual contexts, improving scalability and correlation with human judgments.
Contribution
It presents a novel, fully automated method for creating QA-based nuggets for report evaluation, reducing manual effort and enhancing cross-lingual applicability.
Findings
DoGMaTiQ shows strong correlation with human judgments in cross-lingual report evaluation.
A high-quality LLM nugget generator is crucial for system performance.
The pipeline's rankings are robust to outlier systems.
Abstract
Evaluation of long-form, citation-backed reports has lately received significant attention due to the wide-scale adoption of retrieval-augmented generation (RAG) systems. Core to many evaluation frameworks is the use of atomic facts, or nuggets, to assess a report's coverage of query-relevant information attested in the underlying collection. While nuggets have traditionally been represented as short statements, recent work has used question-answer (QA) representations, enabling fine-grained evaluations that decouple the information need (i.e. the question) from the potentially diverse content that satisfies it (i.e. its answers). A persistent challenge for nugget-based evaluation is the need to manually curate sets of nuggets for each topic in a test collection -- a laborious process that scales poorly to novel information needs. This challenge is acute in cross-lingual settings,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
