ISAAC: Auditing Causal Reasoning in Deep Models for Drug-Target Interaction

TL;DR

ISAAC is a novel post-hoc framework that assesses the causal reasoning of deep models in drug-target interaction prediction by probing their structural sensitivity, revealing discrepancies undetectable by accuracy metrics.

Contribution

Introduces ISAAC, a framework for structural auditing of deep models that uncovers reasoning differences in DTI models beyond standard accuracy evaluation.

Findings

ISAAC reveals about 25% relative differences in reasoning scores across models.

Discrepancies are stable across training and intervention seeds.

Differences are undetectable under conventional accuracy metrics.

Abstract

Deep learning models for drug--target interaction (DTI) prediction often achieve strong benchmark performance without necessarily relying on mechanistically meaningful molecular features, a limitation that standard accuracy-based evaluation cannot detect. We introduce ISAAC (Intervention-based Structural Auditing Approach for Causal Reasoning), a post-hoc framework that evaluates prior-relative structural sensitivity by probing frozen models through matched mechanistic and spurious input-level interventions, independently of predictive accuracy. Applied to three sequence-based DTI architectures on the Davis benchmark, ISAAC reveals approximately 25\% relative differences in reasoning scores across models with comparable AUROC (within around 3\%), stable across training and intervention seeds and two distinct perturbation operators. These discrepancies, undetectable under conventional accuracy metrics, motivate the use of post-hoc structural auditing as a complement to standard performance evaluation in scientific machine learning for molecular modeling.