AGRI-Fidelity: Evaluating the Reliability of Listenable Explanations for Poultry Disease Detection

TL;DR

AGRI-Fidelity is a new evaluation framework that assesses the reliability of listenable explanations in poultry disease detection, addressing issues with existing metrics by reducing artifacts and improving trustworthiness.

Contribution

It introduces a reliability-oriented evaluation method combining cross-model consensus and cyclic permutation, enhancing explanation trustworthiness without needing spatial ground truth.

Findings

Effectively discriminates reliable explanations from artifacts.

Outperforms masking-based metrics in noisy farm environments.

Provides a reliability-aware evaluation across datasets.

Abstract

Existing XAI metrics measure faithfulness for a single model, ignoring model multiplicity where near-optimal classifiers rely on different or spurious acoustic cues. In noisy farm environments, stationary artifacts such as ventilation noise can produce explanations that are faithful yet unreliable, as masking-based metrics fail to penalize redundant shortcuts. We propose AGRI-Fidelity, a reliability-oriented evaluation framework for listenable explanations in poultry disease detection without spatial ground truth. The method combines cross-model consensus with cyclic temporal permutation to construct null distributions and compute a False Discovery Rate (FDR), suppressing stationary artifacts while preserving time-localized bioacoustic markers. Across real and controlled datasets, AGRI-Fidelity effectively provides reliability-aware discrimination for all data points versus masking-based metrics.