Using ScrutinAI for Visual Inspection of DNN Performance in a Medical Use Case

TL;DR

This paper presents ScrutinAI, a visual analytics tool designed to help analyze how variations in expert labeling affect deep neural network performance in medical image classification tasks, enabling better understanding of model weaknesses.

Contribution

The paper introduces ScrutinAI, a novel visual analytics approach for diagnosing the impact of label quality variations on DNN performance in medical imaging.

Findings

ScrutinAI effectively identifies label-related weaknesses in DNN models.

Analysis reveals significant influence of expert label variations on model accuracy.

Tool facilitates root cause analysis distinguishing labeling issues from model deficiencies.

Abstract

Our Visual Analytics (VA) tool ScrutinAI supports human analysts to investigate interactively model performanceand data sets. Model performance depends on labeling quality to a large extent. In particular in medical settings, generation of high quality labels requires in depth expert knowledge and is very costly. Often, data sets are labeled by collecting opinions of groups of experts. We use our VA tool to analyse the influence of label variations between different experts on the model performance. ScrutinAI facilitates to perform a root cause analysis that distinguishes weaknesses of deep neural network (DNN) models caused by varying or missing labeling quality from true weaknesses. We scrutinize the overall detection of intracranial hemorrhages and the more subtle differentiation between subtypes in a publicly available data set.