Long-Form Information Alignment Evaluation Beyond Atomic Facts

TL;DR

This paper introduces DoveScore, a new framework for evaluating long-form information alignment that models inter-fact relationships, outperforming existing methods and addressing vulnerabilities in current evaluators.

Contribution

We propose DoveScore, a novel evaluation framework that jointly verifies factual accuracy and event-order consistency, improving robustness over existing methods.

Findings

DoveScore outperforms existing fine-grained methods by over 8%.

MontageLie benchmark reveals vulnerabilities in current evaluators.

Current evaluators have AUC-ROC scores below 65% on the benchmark.

Abstract

Information alignment evaluators are vital for various NLG evaluation tasks and trustworthy LLM deployment, reducing hallucinations and enhancing user trust. Current fine-grained methods, like FactScore, verify facts individually but neglect inter-fact dependencies, enabling subtle vulnerabilities. In this work, we introduce MontageLie, a challenging benchmark that constructs deceptive narratives by "montaging" truthful statements without introducing explicit hallucinations. We demonstrate that both coarse-grained LLM-based evaluators and current fine-grained frameworks are susceptible to this attack, with AUC-ROC scores falling below 65%. To enable more robust fine-grained evaluation, we propose DoveScore, a novel framework that jointly verifies factual accuracy and event-order consistency. By modeling inter-fact relationships, DoveScore outperforms existing fine-grained methods by over 8%, providing a more robust solution for long-form text alignment evaluation. Our code and datasets are available at https://github.com/dannalily/DoveScore.