SIEFormer: Spectral-Interpretable and -Enhanced Transformer for Generalized Category Discovery

TL;DR

SIEFormer introduces a spectral analysis-based transformer architecture that improves feature adaptability and interpretability for generalized category discovery tasks, achieving state-of-the-art results.

Contribution

The paper proposes a novel spectral-interpretable transformer with implicit and explicit branches, including a Band-adaptive Filter and Maneuverable Filtering Layer, enhancing global and local token dependencies.

Findings

Achieves state-of-the-art performance on multiple image recognition datasets.

Demonstrates the effectiveness of spectral analysis in transformer interpretability.

Validates the approach through extensive ablation studies and visualizations.

Abstract

This paper presents a novel approach, Spectral-Interpretable and -Enhanced Transformer (SIEFormer), which leverages spectral analysis to reinterpret the attention mechanism within Vision Transformer (ViT) and enhance feature adaptability, with particular emphasis on challenging Generalized Category Discovery (GCD) tasks. The proposed SIEFormer is composed of two main branches, each corresponding to an implicit and explicit spectral perspective of the ViT, enabling joint optimization. The implicit branch realizes the use of different types of graph Laplacians to model the local structure correlations of tokens, along with a novel Band-adaptive Filter (BaF) layer that can flexibly perform both band-pass and band-reject filtering. The explicit branch, on the other hand, introduces a Maneuverable Filtering Layer (MFL) that learns global dependencies among tokens by applying the Fourier transform to the input ``value" features, modulating the transformed signal with a set of learnable parameters in the frequency domain, and then performing an inverse Fourier transform to obtain the enhanced features. Extensive experiments reveal state-of-the-art performance on multiple image recognition datasets, reaffirming the superiority of our approach through ablation studies and visualizations.