HMS-VesselNet: Hierarchical Multi-Scale Attention Network with Topology-Preserving Loss for Retinal Vessel Segmentation

TL;DR

HMS-VesselNet is a hierarchical multi-scale neural network with a topology-preserving loss function designed to improve retinal vessel segmentation, especially for thin peripheral vessels, achieving high accuracy and robustness across multiple datasets.

Contribution

The paper introduces a novel multi-scale network architecture with a combined loss function and hard example mining for enhanced vessel segmentation performance.

Findings

Achieves a mean Dice score of 88.72% on multiple datasets.

Significantly improves recall of thin peripheral vessels.

Maintains high AUC (>95%) on unseen datasets.

Abstract

Retinal vessel segmentation methods based on standard overlap losses tend to miss thin peripheral vessels because these structures occupy very few pixels and have low contrast against the background. We propose HMS-VesselNet, a hierarchical multi-scale network that processes fundus images across four parallel branches at different resolutions and combines their outputs using learned fusion weights. The training loss combines Dice, binary cross-entropy, and centerline Dice to jointly optimize area overlap and vessel continuity. Hard example mining is applied from epoch 20 onward to concentrate gradient updates on the most difficult training images. Tested on 68 images from DRIVE, STARE, and CHASE_DB1 using 5-fold cross-validation, the model achieves a mean Dice of 88.72 +/- 0.67%, Sensitivity of 90.78 +/- 1.42%, and AUC of 98.25 +/- 0.21%. In leave-one-dataset-out experiments, AUC remains above 95% on each unseen dataset. The largest improvement is in the recall of thin peripheral vessels, which are the structures most frequently missed by standard methods and most critical for early detection of diabetic retinopathy.