ResFormer: Scaling ViTs with Multi-Resolution Training

TL;DR

ResFormer introduces a multi-resolution training framework for Vision Transformers, enhancing their ability to perform well across a wide range of input resolutions through scale consistency and adaptive positional embeddings.

Contribution

It presents a novel multi-resolution training method with scale consistency loss and a global-local positional embedding strategy for improved resolution scalability in ViTs.

Findings

ResFormer-B-MR achieves 75.86% Top-1 accuracy at 96 resolution.

ResFormer outperforms DeiT-B by 48% at low resolution.

The framework extends effectively to segmentation, detection, and video tasks.

Abstract

Vision Transformers (ViTs) have achieved overwhelming success, yet they suffer from vulnerable resolution scalability, i.e., the performance drops drastically when presented with input resolutions that are unseen during training. We introduce, ResFormer, a framework that is built upon the seminal idea of multi-resolution training for improved performance on a wide spectrum of, mostly unseen, testing resolutions. In particular, ResFormer operates on replicated images of different resolutions and enforces a scale consistency loss to engage interactive information across different scales. More importantly, to alternate among varying resolutions effectively, especially novel ones in testing, we propose a global-local positional embedding strategy that changes smoothly conditioned on input sizes. We conduct extensive experiments for image classification on ImageNet. The results provide strong quantitative evidence that ResFormer has promising scaling abilities towards a wide range of resolutions. For instance, ResFormer-B-MR achieves a Top-1 accuracy of 75.86% and 81.72% when evaluated on relatively low and high resolutions respectively (i.e., 96 and 640), which are 48% and 7.49% better than DeiT-B. We also demonstrate, moreover, ResFormer is flexible and can be easily extended to semantic segmentation, object detection and video action recognition. Code is available at https://github.com/ruitian12/resformer.