Disentangled representations: towards interpretation of sex determination from hip bone

TL;DR

This paper introduces a disentangled variational auto-encoder approach to interpret neural network decisions in medical imaging, specifically for automatic sex determination from hip bones, by providing interpretable class-specific features.

Contribution

It proposes a novel interpretability paradigm using disentangled VAEs that explicitly represent class information and enable sample transformation between classes.

Findings

Features identified align with expert knowledge.

Model successfully distinguishes sex from hip bones.

Enables visualization of class differences through sample transformation.

Abstract

By highlighting the regions of the input image that contribute the most to the decision, saliency maps have become a popular method to make neural networks interpretable. In medical imaging, they are particularly well-suited to explain neural networks in the context of abnormality localization. However, from our experiments, they are less suited to classification problems where the features that allow to distinguish between the different classes are spatially correlated, scattered and definitely non-trivial. In this paper we thus propose a new paradigm for better interpretability. To this end we provide the user with relevant and easily interpretable information so that he can form his own opinion. We use Disentangled Variational Auto-Encoders which latent representation is divided into two components: the non-interpretable part and the disentangled part. The latter accounts for the categorical variables explicitly representing the different classes of interest. In addition to providing the class of a given input sample, such a model offers the possibility to transform the sample from a given class to a sample of another class, by modifying the value of the categorical variables in the latent representation. This paves the way to easier interpretation of class differences. We illustrate the relevance of this approach in the context of automatic sex determination from hip bones in forensic medicine. The features encoded by the model, that distinguish the different classes were found to be consistent with expert knowledge.