Interpretable Vision Transformers in Monocular Depth Estimation via SVDA

TL;DR

This paper introduces SVDA, a spectrally structured attention mechanism for vision transformers in monocular depth estimation, enhancing interpretability while maintaining accuracy and providing new insights into attention organization.

Contribution

The paper presents SVDA, a novel spectrally structured attention formulation that makes transformer attention interpretable in dense prediction tasks like depth estimation.

Findings

SVDA preserves or slightly improves accuracy on KITTI and NYU-v2 datasets.

SVDA provides six spectral indicators revealing attention organization patterns.

Attention interpretability is significantly enhanced with minimal computational overhead.

Abstract

Monocular depth estimation is a central problem in computer vision with applications in robotics, AR, and autonomous driving, yet the self-attention mechanisms that drive modern Transformer architectures remain opaque. We introduce SVD-Inspired Attention (SVDA) into the Dense Prediction Transformer (DPT), providing the first spectrally structured formulation of attention for dense prediction tasks. SVDA decouples directional alignment from spectral modulation by embedding a learnable diagonal matrix into normalized query-key interactions, enabling attention maps that are intrinsically interpretable rather than post-hoc approximations. Experiments on KITTI and NYU-v2 show that SVDA preserves or slightly improves predictive accuracy while adding only minor computational overhead. More importantly, SVDA unlocks six spectral indicators that quantify entropy, rank, sparsity, alignment, selectivity, and robustness. These reveal consistent cross-dataset and depth-wise patterns in how attention organizes during training, insights that remain inaccessible in standard Transformers. By shifting the role of attention from opaque mechanism to quantifiable descriptor, SVDA redefines interpretability in monocular depth estimation and opens a principled avenue toward transparent dense prediction models.