Beyond True or False: Retrieval-Augmented Hierarchical Analysis of Nuanced Claims

TL;DR

This paper introduces ClaimSpect, a retrieval-augmented hierarchical framework that dissects nuanced claims into aspects and perspectives, enabling structured analysis and validation using relevant corpus segments.

Contribution

ClaimSpect is a novel retrieval-augmented generation framework that hierarchically decomposes claims into aspects and perspectives, improving nuanced claim analysis.

Findings

Effective in dissecting complex claims into aspects and perspectives

Accurately retrieves relevant corpus segments for validation

Outperforms multiple baseline methods in real-world case studies

Abstract

Claims made by individuals or entities are oftentimes nuanced and cannot be clearly labeled as entirely "true" or "false" -- as is frequently the case with scientific and political claims. However, a claim (e.g., "vaccine A is better than vaccine B") can be dissected into its integral aspects and sub-aspects (e.g., efficacy, safety, distribution), which are individually easier to validate. This enables a more comprehensive, structured response that provides a well-rounded perspective on a given problem while also allowing the reader to prioritize specific angles of interest within the claim (e.g., safety towards children). Thus, we propose ClaimSpect, a retrieval-augmented generation-based framework for automatically constructing a hierarchy of aspects typically considered when addressing a claim and enriching them with corpus-specific perspectives. This structure hierarchically partitions an input corpus to retrieve relevant segments, which assist in discovering new sub-aspects. Moreover, these segments enable the discovery of varying perspectives towards an aspect of the claim (e.g., support, neutral, or oppose) and their respective prevalence (e.g., "how many biomedical papers believe vaccine A is more transportable than B?"). We apply ClaimSpect to a wide variety of real-world scientific and political claims featured in our constructed dataset, showcasing its robustness and accuracy in deconstructing a nuanced claim and representing perspectives within a corpus. Through real-world case studies and human evaluation, we validate its effectiveness over multiple baselines.