Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?

TL;DR

This paper introduces SimPool, a universal attention-based pooling method that enhances supervised and self-supervised vision transformers by improving performance and generating high-quality attention maps without architectural modifications.

Contribution

The paper develops a generic pooling framework and proposes SimPool, a simple attention-based pooling mechanism that improves transformer performance and attention map quality across supervision types.

Findings

SimPool improves performance on pre-training and downstream tasks.

SimPool generates high-quality attention maps in supervised transformers.

SimPool is effective for both convolutional and transformer encoders.

Abstract

Convolutional networks and vision transformers have different forms of pairwise interactions, pooling across layers and pooling at the end of the network. Does the latter really need to be different? As a by-product of pooling, vision transformers provide spatial attention for free, but this is most often of low quality unless self-supervised, which is not well studied. Is supervision really the problem? In this work, we develop a generic pooling framework and then we formulate a number of existing methods as instantiations. By discussing the properties of each group of methods, we derive SimPool, a simple attention-based pooling mechanism as a replacement of the default one for both convolutional and transformer encoders. We find that, whether supervised or self-supervised, this improves performance on pre-training and downstream tasks and provides attention maps delineating object boundaries in all cases. One could thus call SimPool universal. To our knowledge, we are the first to obtain attention maps in supervised transformers of at least as good quality as self-supervised, without explicit losses or modifying the architecture. Code at: https://github.com/billpsomas/simpool.