Prompted Zero-Shot Multi-label Classification of Factual Incorrectness in Machine-Generated Summaries

TL;DR

This paper presents a prompt-based classification system to identify and categorize factual inaccuracies in machine-generated summaries, aiming to improve the detection of errors like misrepresentation and fabrication.

Contribution

Introduces a novel prompt-based approach for classifying factual errors in summaries, categorizing errors into four specific types for better error analysis.

Findings

Prompt-based methods can detect factual errors to some extent.

Errors include misrepresentation, incorrect measurements, false attribution, and fabrication.

Scope for improving classification accuracy.

Abstract

This study addresses the critical issue of factual inaccuracies in machine-generated text summaries, an increasingly prevalent issue in information dissemination. Recognizing the potential of such errors to compromise information reliability, we investigate the nature of factual inconsistencies across machine-summarized content. We introduce a prompt-based classification system that categorizes errors into four distinct types: misrepresentation, inaccurate quantities or measurements, false attribution, and fabrication. The participants are tasked with evaluating a corpus of machine-generated summaries against their original articles. Our methodology employs qualitative judgements to identify the occurrence of factual distortions. The results show that our prompt-based approaches are able to detect the type of errors in the summaries to some extent, although there is scope for improvement in our classification systems.