Evaluation of Faithfulness Using the Longest Supported Subsequence

TL;DR

This paper introduces a new metric called Longest Supported Subsequence (LSS) to evaluate the faithfulness of machine-generated text, demonstrating improved correlation with human judgments and outperforming existing metrics across multiple datasets and models.

Contribution

The paper proposes a novel LSS-based metric for faithfulness evaluation, supported by a new dataset and a fine-tuned model, enhancing accuracy over previous methods.

Findings

LSS metric correlates better with human ratings.

LSS improves faithfulness evaluation by 18% over state-of-the-art.

The metric outperforms others across six summarization models.

Abstract

As increasingly sophisticated language models emerge, their trustworthiness becomes a pivotal issue, especially in tasks such as summarization and question-answering. Ensuring their responses are contextually grounded and faithful is challenging due to the linguistic diversity and the myriad of possible answers. In this paper, we introduce a novel approach to evaluate faithfulness of machine-generated text by computing the longest noncontinuous substring of the claim that is supported by the context, which we refer to as the Longest Supported Subsequence (LSS). Using a new human-annotated dataset, we finetune a model to generate LSS. We introduce a new method of evaluation and demonstrate that these metrics correlate better with human ratings when LSS is employed, as opposed to when it is not. Our proposed metric demonstrates an 18% enhancement over the prevailing state-of-the-art metric for faithfulness on our dataset. Our metric consistently outperforms other metrics on a summarization dataset across six different models. Finally, we compare several popular Large Language Models (LLMs) for faithfulness using this metric. We release the human-annotated dataset built for predicting LSS and our fine-tuned model for evaluating faithfulness.