AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment

TL;DR

AlignCheck introduces an interpretable, schema-free, and weighted factual consistency metric for open-domain texts, effectively addressing hallucination issues in large language models, especially in high-stakes domains like clinical applications.

Contribution

It presents a novel, flexible framework that decomposes text into atomic facts and incorporates a weighted metric for improved factual assessment, with mechanisms to control complexity.

Findings

Benchmarking on general and clinical datasets shows improved factual accuracy assessment.

The method enhances interpretability and flexibility over existing metrics.

Code release supports future fact-aware model training.

Abstract

Large Language Models have significantly advanced natural language processing tasks, but remain prone to generating incorrect or misleading but plausible arguments. This issue, known as hallucination, is particularly concerning in high-stakes domains like clinical applications, where factual inaccuracies can have severe consequences. Existing evaluation metrics fail to adequately assess factual consistency and lack interpretability, making diagnosing and mitigating errors difficult. We propose an interpretable framework for factual consistency assessment for in-domain and open-domain texts to address these limitations. Our approach decomposes text into atomic facts and introduces a flexible, schema-free methodology. Unlike previous methods with an absolute metric, we incorporate a weighted metric to enhance factual evaluation. Additionally, we propose a mechanism to control assessment complexity in intricate domains. We benchmark our approach on popular general and clinical datasets and release our code to support fact-aware model training in future research.