Interpretable Vision Transformers in Image Classification via SVDA

TL;DR

This paper introduces SVDA, a geometrically grounded attention mechanism for Vision Transformers that improves interpretability and spectral structure without reducing classification performance across multiple benchmarks.

Contribution

We adapt SVDA to Vision Transformers, enhancing interpretability and spectral structure while maintaining accuracy, and demonstrate its effectiveness on standard image classification benchmarks.

Findings

SVDA produces more interpretable attention patterns.

SVDA maintains classification accuracy across benchmarks.

SVDA provides spectral diagnostics for attention models.

Abstract

Vision Transformers (ViTs) have achieved state-of-the-art performance in image classification, yet their attention mechanisms often remain opaque and exhibit dense, non-structured behaviors. In this work, we adapt our previously proposed SVD-Inspired Attention (SVDA) mechanism to the ViT architecture, introducing a geometrically grounded formulation that enhances interpretability, sparsity, and spectral structure. We apply the use of interpretability indicators -- originally proposed with SVDA -- to monitor attention dynamics during training and assess structural properties of the learned representations. Experimental evaluations on four widely used benchmarks -- CIFAR-10, FashionMNIST, CIFAR-100, and ImageNet-100 -- demonstrate that SVDA consistently yields more interpretable attention patterns without sacrificing classification accuracy. While the current framework offers descriptive insights rather than prescriptive guidance, our results establish SVDA as a comprehensive and informative tool for analyzing and developing structured attention models in computer vision. This work lays the foundation for future advances in explainable AI, spectral diagnostics, and attention-based model compression.