Sparse Multi-Modal Transformer with Masking for Alzheimer's Disease Classification

TL;DR

This paper introduces SMMT, a sparse multi-modal transformer that reduces computational costs and improves robustness for Alzheimer's Disease classification, maintaining performance while being more resource-efficient.

Contribution

The paper proposes a novel sparse attention mechanism and modality-wise masking in a multi-modal transformer, enhancing efficiency and robustness for resource-constrained applications.

Findings

Significantly reduces training time, memory, and energy consumption.

Maintains competitive accuracy in Alzheimer's classification.

Demonstrates scalability and robustness with incomplete inputs.

Abstract

Transformer-based multi-modal intelligent systems often suffer from high computational and energy costs due to dense self-attention, limiting their scalability under resource constraints. This paper presents SMMT, a sparse multi-modal transformer architecture designed to improve efficiency and robustness. Building upon a cascaded multi-modal transformer framework, SMMT introduces cluster-based sparse attention to achieve near linear computational complexity and modality-wise masking to enhance robustness against incomplete inputs. The architecture is evaluated using Alzheimer's Disease classification on the ADNI dataset as a representative multi-modal case study. Experimental results show that SMMT maintains competitive predictive performance while significantly reducing training time, memory usage, and energy consumption compared to dense attention baselines, demonstrating its suitability as a resource-aware architectural component for scalable intelligent systems.