Knowledge-based anomaly detection for identifying network-induced shape artifacts

TL;DR

This paper presents a novel two-stage knowledge-based anomaly detection method that effectively identifies network-induced shape artifacts in synthetic medical images, improving quality assessment for synthetic data used in machine learning.

Contribution

The work introduces a new feature extraction technique and an anomaly detection framework specifically designed for detecting shape artifacts in synthetic images, with demonstrated high accuracy and agreement with human experts.

Findings

Achieved AUC of 0.97 and 0.91 in artifact detection.

Human reader agreement rates of 66% and 68% for most anomalous images.

Kendall-Tau correlations of 0.45 and 0.43 indicating reasonable alignment with human judgment.

Abstract

Synthetic data provides a promising approach to address data scarcity for training machine learning models; however, adoption without proper quality assessments may introduce artifacts, distortions, and unrealistic features that compromise model performance and clinical utility. This work introduces a novel knowledge-based anomaly detection method for detecting network-induced shape artifacts in synthetic images. The introduced method utilizes a two-stage framework comprising (i) a novel feature extractor that constructs a specialized feature space by analyzing the per-image distribution of angle gradients along anatomical boundaries, and (ii) an isolation forest-based anomaly detector. We demonstrate the effectiveness of the method for identifying network-induced shape artifacts in two synthetic mammography datasets from models trained on CSAW-M and VinDr-Mammo patient datasets respectively. Quantitative evaluation shows that the method successfully concentrates artifacts in the most anomalous partition (1st percentile), with AUC values of 0.97 (CSAW-syn) and 0.91 (VMLO-syn). In addition, a reader study involving three imaging scientists confirmed that images identified by the method as containing network-induced shape artifacts were also flagged by human readers with mean agreement rates of 66% (CSAW-syn) and 68% (VMLO-syn) for the most anomalous partition, approximately 1.5-2 times higher than the least anomalous partition. Kendall-Tau correlations between algorithmic and human rankings were 0.45 and 0.43 for the two datasets, indicating reasonable agreement despite the challenging nature of subtle artifact detection. This method is a step forward in the responsible use of synthetic data, as it allows developers to evaluate synthetic images for known anatomic constraints and pinpoint and address specific issues to improve the overall quality of a synthetic dataset.