Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers

TL;DR

This paper introduces DCT-based methods to improve the initialization and efficiency of Vision Transformers, leading to better accuracy and reduced computation without sacrificing performance.

Contribution

It proposes novel DCT-based initialization and attention compression techniques that enhance Vision Transformer training and efficiency.

Findings

DCT-based initialization improves classification accuracy on CIFAR-10 and ImageNet-1K.

DCT-based attention compression reduces computational overhead significantly.

High-frequency DCT component truncation maintains accuracy while decreasing complexity.

Abstract

Self-attention is central to the success of Transformer architectures; however, learning the query, key, and value projections from random initialization remains challenging and computationally expensive. In this paper, we propose two complementary methods that leverage the Discrete Cosine Transform (DCT) to enhance the efficiency and performance of Vision Transformers. First, we address the initialization problem by introducing a simple yet effective DCT-based initialization strategy for self-attention, where projection weights are initialized using DCT coefficients. This structure-preserving approach consistently improves classification accuracy on the CIFAR-10 and ImageNet-1K benchmarks. Second, we propose a DCT-based attention compression technique that exploits the decorrelation properties of the frequency domain. By observing that high-frequency DCT coefficients typically correspond to noise, we truncate high-frequency components of the input patches, thereby reducing the dimensionality of the query, key, and value projections without sacrificing accuracy. Experiments on Swin Transformer models demonstrate that the proposed compression method achieves a substantial reduction in computational overhead while maintaining comparable performance.