MedScore: Generalizable Factuality Evaluation of Free-Form Medical Answers by Domain-adapted Claim Decomposition and Verification

TL;DR

MedScore is a domain-adapted, modular pipeline that improves factuality evaluation of free-form medical answers by better decomposing claims and verifying them against in-domain data, reducing hallucinations.

Contribution

It introduces MedScore, a novel, domain-specific factuality evaluation pipeline that enhances claim decomposition and verification for medical answers, outperforming existing methods.

Findings

Extracts up to three times more valid facts than previous methods.

Reduces hallucination and vague references in medical answer evaluation.

Factuality scores vary significantly with different decomposition and verification methods.

Abstract

While Large Language Models (LLMs) can generate fluent and convincing responses, they are not necessarily correct. This is especially apparent in the popular decompose-then-verify factuality evaluation pipeline, where LLMs evaluate generations by decomposing the generations into individual, valid claims. Factuality evaluation is especially important for medical answers, since incorrect medical information could seriously harm the patient. However, existing factuality systems are a poor match for the medical domain, as they are typically only evaluated on objective, entity-centric, formulaic texts such as biographies and historical topics. This differs from condition-dependent, conversational, hypothetical, sentence-structure diverse, and subjective medical answers, which makes decomposition into valid facts challenging. We propose MedScore, a new pipeline to decompose medical answers into condition-aware valid facts and verify against in-domain corpora. Our method extracts up to three times more valid facts than existing methods, reducing hallucination and vague references, and retaining condition-dependency in facts. The resulting factuality score substantially varies by decomposition method, verification corpus, and used backbone LLM, highlighting the importance of customizing each step for reliable factuality evaluation by using our generalizable and modularized pipeline for domain adaptation.