Towards a perturbation-based explanation for medical AI as differentiable programs

TL;DR

This paper proposes a perturbation-based explanation method for medical AI models using Jacobian matrices to assess model stability against input changes, aiming to improve interpretability in clinical settings.

Contribution

It introduces a novel approach to explain AI models in medicine by analyzing their Jacobian matrices, independent of training data or feature importance biases.

Findings

Jacobian matrices can measure model response stability to input perturbations.

Preliminary results show feasibility of using Jacobian for model explanation.

This approach offers an objective, data-independent explanation method.

Abstract

Recent advancement in machine learning algorithms reaches a point where medical devices can be equipped with artificial intelligence (AI) models for diagnostic support and routine automation in clinical settings. In medicine and healthcare, there is a particular demand for sufficient and objective explainability of the outcome generated by AI models. However, AI models are generally considered as black boxes due to their complexity, and the computational process leading to their response is often opaque. Although several methods have been proposed to explain the behavior of models by evaluating the importance of each feature in discrimination and prediction, they may suffer from biases and opacities arising from the scale and sampling protocol of the dataset used for training or testing. To overcome the shortcomings of existing methods, we explore an alternative approach to provide an objective explanation of AI models that can be defined independently of the learning process and does not require additional data. As a preliminary study for this direction of research, this work examines a numerical availability of the Jacobian matrix of deep learning models that measures how stably a model responses against small perturbations added to the input. The indicator, if available, are calculated from a trained AI model for a given target input. This is a first step towards a perturbation-based explanation, which will assist medical practitioners in understanding and interpreting the response of the AI model in its clinical application.