Cross-Examination Framework: A Task-Agnostic Diagnostic for Information Fidelity in Text-to-Text Generation

TL;DR

The paper introduces the Cross-Examination Framework (CEF), a reference-free, multi-dimensional evaluation method for text-to-text generation that effectively identifies semantic errors and correlates well with traditional metrics.

Contribution

It presents a novel, task-agnostic diagnostic framework that evaluates semantic fidelity without gold references, validated across multiple generation tasks and with human expert validation.

Findings

CEF accurately detects content omissions and factual contradictions.

Strong correlation between reference-free and with-reference modes validates CEF.

Human validation shows CEF effectively identifies semantic errors, especially entity and relation distortions.

Abstract

Traditional metrics like BLEU and BERTScore fail to capture semantic fidelity in generative text-to-text tasks. We adapt the Cross-Examination Framework (CEF) for a reference-free, multi-dimensional evaluation by treating the source and candidate as independent knowledge bases. CEF generates verifiable questions from each text and performs a cross-examination to derive three interpretable scores: Coverage, Conformity, and Consistency. Validated across translation, summarization and clinical note-generation, our framework identifies critical errors, such as content omissions and factual contradictions, missed by standard metrics. A key contribution is a systematic robustness analysis to select a stable judge model. Crucially, the strong correlation between our reference-free and with-reference modes validates CEF's reliability without gold references. Furthermore, human expert validation demonstrates that CEF mismatching questions align with meaning-altering semantic errors higher than with non-semantic errors, particularly excelling at identifying entity-based and relational distortions.