Focal Attention Networks: optimising attention for biomedical image segmentation

TL;DR

This paper explores the role of the Focal parameter in attention mechanisms for biomedical image segmentation, proposing a unified loss framework and heuristics to optimize attention module integration for improved performance and efficiency.

Contribution

It introduces a Focal distance penalty in the loss function, extending the Unified Focal loss, and provides heuristics for optimal attention module placement in CNNs.

Findings

Focal parameter modulates attention effectiveness.

Incorporating boundary-based losses improves segmentation.

Optimal attention use enhances performance with fewer modules.

Abstract

In recent years, there has been increasing interest to incorporate attention into deep learning architectures for biomedical image segmentation. The modular design of attention mechanisms enables flexible integration into convolutional neural network architectures, such as the U-Net. Whether attention is appropriate to use, what type of attention to use, and where in the network to incorporate attention modules, are all important considerations that are currently overlooked. In this paper, we investigate the role of the Focal parameter in modulating attention, revealing a link between attention in loss functions and networks. By incorporating a Focal distance penalty term, we extend the Unified Focal loss framework to include boundary-based losses. Furthermore, we develop a simple and interpretable, dataset and model-specific heuristic to integrate the Focal parameter into the Squeeze-and-Excitation block and Attention Gate, achieving optimal performance with fewer number of attention modules on three well-validated biomedical imaging datasets, suggesting judicious use of attention modules results in better performance and efficiency.