Hard-Attention Gates with Gradient Routing for Endoscopic Image Computing

TL;DR

This paper introduces Hard-Attention Gates with Gradient Routing to improve feature selection in CNNs and ViTs, significantly enhancing polyp size classification accuracy in endoscopic images and reducing overfitting.

Contribution

It presents a novel combination of Hard-Attention Gates and Gradient Routing for dynamic feature selection, regularization, and improved generalization in medical image analysis.

Findings

F1 score for binary classification reached 87.8% with HAG-enhanced CNNs.

ViT-T model achieved 76.5% F1 score in triclass polyp sizing.

Codebase release facilitates standardized evaluation on endoscopic datasets.

Abstract

To address overfitting and enhance model generalization in gastroenterological polyp size assessment, our study introduces Feature-Selection Gates (FSG) or Hard-Attention Gates (HAG) alongside Gradient Routing (GR) for dynamic feature selection. This technique aims to boost Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) by promoting sparse connectivity, thereby reducing overfitting and enhancing generalization. HAG achieves this through sparsification with learnable weights, serving as a regularization strategy. GR further refines this process by optimizing HAG parameters via dual forward passes, independently from the main model, to improve feature re-weighting. Our evaluation spanned multiple datasets, including CIFAR-100 for a broad impact assessment and specialized endoscopic datasets (REAL-Colon, Misawa, and SUN) focusing on polyp size estimation, covering over 200 polyps in more than 370,000 frames. The findings indicate that our HAG-enhanced networks substantially enhance performance in both binary and triclass classification tasks related to polyp sizing. Specifically, CNNs experienced an F1 Score improvement to 87.8% in binary classification, while in triclass classification, the ViT-T model reached an F1 Score of 76.5%, outperforming traditional CNNs and ViT-T models. To facilitate further research, we are releasing our codebase, which includes implementations for CNNs, multistream CNNs, ViT, and HAG-augmented variants. This resource aims to standardize the use of endoscopic datasets, providing public training-validation-testing splits for reliable and comparable research in gastroenterological polyp size estimation. The codebase is available at github.com/cosmoimd/feature-selection-gates.