HAViT: Historical Attention Vision Transformer

TL;DR

HAViT introduces a cross-layer attention propagation method for Vision Transformers, leveraging historical attention matrices to improve feature learning and accuracy with minimal architectural changes.

Contribution

The paper proposes a novel cross-layer attention propagation technique that refines inter-layer information flow in Vision Transformers, enhancing performance across multiple datasets and architectures.

Findings

Consistent accuracy improvements on CIFAR-100 and TinyImageNet.

Optimal blending hyperparameter identified at alpha=0.45.

Random initialization outperforms zero initialization for attention matrices.

Abstract

Vision Transformers have excelled in computer vision but their attention mechanisms operate independently across layers, limiting information flow and feature learning. We propose an effective cross-layer attention propagation method that preserves and integrates historical attention matrices across encoder layers, offering a principled refinement of inter-layer information flow in Vision Transformers. This approach enables progressive refinement of attention patterns throughout the transformer hierarchy, enhancing feature acquisition and optimization dynamics. The method requires minimal architectural changes, adding only attention matrix storage and blending operations. Comprehensive experiments on CIFAR-100 and TinyImageNet demonstrate consistent accuracy improvements, with ViT performance increasing from 75.74% to 77.07% on CIFAR-100 (+1.33%) and from 57.82% to 59.07% on TinyImageNet (+1.25%). Cross-architecture validation shows similar gains across transformer variants, with CaiT showing 1.01% enhancement. Systematic analysis identifies the blending hyperparameter of historical attention (alpha = 0.45) as optimal across all configurations, providing the ideal balance between current and historical attention information. Random initialization consistently outperforms zero initialization, indicating that diverse initial attention patterns accelerate convergence and improve final performance. Our code is publicly available at https://github.com/banik-s/HAViT.