Revisiting Transformers with Insights from Image Filtering and Boosting

TL;DR

This paper develops a unifying image processing framework to interpret self-attention in transformers, explaining architectural components and demonstrating that image filtering insights can improve accuracy and robustness in language and vision tasks.

Contribution

It introduces a comprehensive framework for understanding self-attention and its components, and proposes architectural modifications inspired by image processing that enhance transformer performance.

Findings

Framework explains self-attention and its components.

Image processing-inspired modifications improve accuracy and robustness.

Enhanced understanding aids in designing better transformer architectures.

Abstract

The self-attention mechanism, a cornerstone of Transformer-based state-of-the-art deep learning architectures, is largely heuristic-driven and fundamentally challenging to interpret. Establishing a robust theoretical foundation to explain its remarkable success and limitations has therefore become an increasingly prominent focus in recent research. Some notable directions have explored understanding self-attention through the lens of image denoising and nonparametric regression. While promising, existing frameworks still lack a deeper mechanistic interpretation of various architectural components that enhance self-attention, both in its original formulation and subsequent variants. In this work, we aim to advance this understanding by developing a unifying image processing framework, capable of explaining not only the self-attention computation itself but also the role of components such as positional encoding and residual connections, including numerous later variants. We also pinpoint potential distinctions between the two concepts building upon our framework, and make effort to close this gap. We introduce two independent architectural modifications within transformers. While our primary objective is interpretability, we empirically observe that image processing-inspired modifications can also lead to notably improved accuracy and robustness against data contamination and adversaries across language and vision tasks as well as better long sequence understanding.