Vision Transformers in 2022: An Update on Tiny ImageNet

TL;DR

This paper evaluates the performance of various vision transformer models on Tiny ImageNet, highlighting Swin Transformer's superior accuracy and providing an update on their transfer learning capabilities on smaller datasets.

Contribution

It offers the first comprehensive update on vision transformers' performance on Tiny ImageNet, including new results for Swin Transformers and other models.

Findings

Swin Transformers achieve 91.35% accuracy on Tiny ImageNet.

Vision transformers perform competitively on small datasets after fine-tuning.

The paper provides code for reproducibility.

Abstract

The recent advances in image transformers have shown impressive results and have largely closed the gap between traditional CNN architectures. The standard procedure is to train on large datasets like ImageNet-21k and then finetune on ImageNet-1k. After finetuning, researches will often consider the transfer learning performance on smaller datasets such as CIFAR-10/100 but have left out Tiny ImageNet. This paper offers an update on vision transformers' performance on Tiny ImageNet. I include Vision Transformer (ViT) , Data Efficient Image Transformer (DeiT), Class Attention in Image Transformer (CaiT), and Swin Transformers. In addition, Swin Transformers beats the current state-of-the-art result with a validation accuracy of 91.35%. Code is available here: https://github.com/ehuynh1106/TinyImageNet-Transformers