Learning to Seek Evidence: A Verifiable Reasoning Agent with Causal Faithfulness Analysis

TL;DR

This paper introduces an interactive, reinforcement learning-based reasoning agent that seeks external visual evidence to produce verifiable explanations, significantly improving diagnostic accuracy and faithfulness in high-stakes AI applications.

Contribution

It presents a novel approach combining evidence-seeking behavior with causal faithfulness analysis to enhance verifiability and trustworthiness of AI explanations.

Findings

Reduced Brier score by 18% with evidence-seeking

Causal intervention confirms evidence importance

Improved calibration and generalization

Abstract

Explanations for AI models in high-stakes domains like medicine often lack verifiability, which can hinder trust. To address this, we propose an interactive agent that produces explanations through an auditable sequence of actions. The agent learns a policy to strategically seek external visual evidence to support its diagnostic reasoning. This policy is optimized using reinforcement learning, resulting in a model that is both efficient and generalizable. Our experiments show that this action-based reasoning process significantly improves calibrated accuracy, reducing the Brier score by 18\% compared to a non-interactive baseline. To validate the faithfulness of the agent's explanations, we introduce a causal intervention method. By masking the visual evidence the agent chooses to use, we observe a measurable degradation in its performance ($\Delta$Brier=+0.029), confirming that the evidence is integral to its decision-making process. Our work provides a practical framework for building AI systems with verifiable and faithful reasoning capabilities.