EG-SpikeFormer: Eye-Gaze Guided Transformer on Spiking Neural Networks for Medical Image Analysis

TL;DR

EG-SpikeFormer is a neuromorphic SNN architecture that uses eye-gaze data to improve medical image analysis, addressing shortcut learning and enhancing interpretability, efficiency, and clinical relevance.

Contribution

This paper introduces EG-SpikeFormer, a novel eye-gaze guided SNN architecture for medical imaging, pioneering neuromorphic computing applications in healthcare.

Findings

Outperforms conventional models in energy efficiency and accuracy

Reduces shortcut learning and improves interpretability

Enhances generalization with multi-modal data alignment

Abstract

Neuromorphic computing has emerged as a promising energy-efficient alternative to traditional artificial intelligence, predominantly utilizing spiking neural networks (SNNs) implemented on neuromorphic hardware. Significant advancements have been made in SNN-based convolutional neural networks (CNNs) and Transformer architectures. However, neuromorphic computing for the medical imaging domain remains underexplored. In this study, we introduce EG-SpikeFormer, an SNN architecture tailored for clinical tasks that incorporates eye-gaze data to guide the model's attention to the diagnostically relevant regions in medical images. Our developed approach effectively addresses shortcut learning issues commonly observed in conventional models, especially in scenarios with limited clinical data and high demands for model reliability, generalizability, and transparency. Our EG-SpikeFormer not only demonstrates superior energy efficiency and performance in medical image prediction tasks but also enhances clinical relevance through multi-modal information alignment. By incorporating eye-gaze data, the model improves interpretability and generalization, opening new directions for applying neuromorphic computing in healthcare.