On the Factual Consistency of Text-based Explainable Recommendation Models

TL;DR

This paper introduces a framework to evaluate the factual consistency of text-based explainable recommendation models, revealing a significant gap between semantic similarity and factual accuracy.

Contribution

It proposes a prompting-based pipeline for extracting factual statements and new metrics for assessing explanation factuality, addressing a critical gap in current evaluation methods.

Findings

Models show high semantic similarity scores but low factuality accuracy.

The proposed metrics reveal a significant gap in explanation factuality.

Benchmark datasets and evaluation pipeline are provided for future research.

Abstract

Text-based explainable recommendation aims to generate natural-language explanations that justify item recommendations, to improve user trust and system transparency. Although recent advances leverage LLMs to produce fluent outputs, a critical question remains underexplored: are these explanations factually consistent with the available evidence? We introduce a comprehensive framework for evaluating the factual consistency of text-based explainable recommenders. We design a prompting-based pipeline that uses LLMs to extract atomic explanatory statements from reviews, thereby constructing a ground truth that isolates and focuses on their factual content. Applying this pipeline to five categories from the Amazon Reviews dataset, we create augmented benchmarks for fine-grained evaluation of explanation quality. We further propose statement-level alignment metrics that combine LLM- and NLI-based approaches to assess both factual consistency and relevance of generated explanations. Across extensive experiments on six state-of-the-art explainable recommendation models, we uncover a critical gap: while models achieve high semantic similarity scores (BERTScore F1: 0.81-0.90), all our factuality metrics reveal alarmingly low performance (LLM-based statement-level precision: 4.38%-32.88%). These findings underscore the need for factuality-aware evaluation in explainable recommendation and provide a foundation for developing more trustworthy explanation systems.