Performance of a deep learning system for detection of referable diabetic retinopathy in real clinical settings

TL;DR

This study evaluates a deep learning system's effectiveness in detecting referable diabetic retinopathy in routine clinical fundus images, demonstrating high accuracy and significant workload reduction potential.

Contribution

The paper presents validation of a commercial AI system for diabetic retinopathy detection in real clinical settings, showing its high accuracy and efficiency in reducing screening workload.

Findings

AUC of 0.988 for DR detection

Sensitivity of 90.53% and specificity of 97.13%

96% workload reduction with minimal false negatives

Abstract

Background: To determine the ability of a commercially available deep learning system, RetCAD v.1.3.1 (Thirona, Nijmegen, The Netherlands) for the automatic detection of referable diabetic retinopathy (DR) on a dataset of colour fundus images acquired during routine clinical practice in a tertiary hospital screening program, analyzing the reduction of workload that can be released incorporating this artificial intelligence-based technology. Methods: Evaluation of the software was performed on a dataset of 7195 nonmydriatic fundus images from 6325 eyes of 3189 diabetic patients attending our screening program between February to December of 2019. The software generated a DR severity score for each colour fundus image which was combined into an eye-level score. This score was then compared with a reference standard as set by a human expert using receiver operating characteristic (ROC) curve analysis. Results: The artificial intelligence (AI) software achieved an area under the ROC curve (AUC) value of 0.988 [0.981:0.993] for the detection of referable DR. At the proposed operating point, the sensitivity of the RetCAD software for DR is 90.53% and specificity is 97.13%. A workload reduction of 96% could be achieved at the cost of only 6 false negatives. Conclusions: The AI software correctly identified the vast majority of referable DR cases, with a workload reduction of 96% of the cases that would need to be checked, while missing almost no true cases, so it may therefore be used as an instrument for triage.