Do We Need Reformer for Vision? An Experimental Comparison with Vision Transformers

TL;DR

This paper compares Reformer and Vision Transformers for computer vision tasks, finding that while Reformers are theoretically more efficient, Vision Transformers outperform them in practical accuracy and efficiency on larger, high-resolution datasets.

Contribution

It provides an experimental comparison of Reformer and Vision Transformers in vision tasks, highlighting the practical limitations of LSH attention for typical image sizes.

Findings

Reformer achieves higher accuracy on CIFAR-10.

Vision Transformers outperform Reformer in efficiency on larger datasets.

LSH attention's theoretical benefits are limited for standard image resolutions.

Abstract

Transformers have recently demonstrated strong performance in computer vision, with Vision Transformers (ViTs) leveraging self-attention to capture both low-level and high-level image features. However, standard ViTs remain computationally expensive, since global self-attention scales quadratically with the number of tokens, which limits their practicality for high-resolution inputs and resource-constrained settings. In this work, we investigate the Reformer architecture as an alternative vision backbone. By combining patch-based tokenization with locality-sensitive hashing (LSH) attention, our model approximates global self-attention while reducing its theoretical time complexity from $\mathcal{O}(n^2)$ to $\mathcal{O}(n \log n)$ in the sequence length $n$. We evaluate the proposed Reformer-based vision model on CIFAR-10 to assess its behavior on small-scale datasets, on ImageNet-100 to study its accuracy--efficiency trade-off in a more realistic setting, and on a high-resolution medical imaging dataset to evaluate the model under longer token sequences. While the Reformer achieves higher accuracy on CIFAR-10 compared to our ViT-style baseline, the ViT model consistently outperforms the Reformer in our experiments in terms of practical efficiency and end-to-end computation time across the larger and higher-resolution settings. These results suggest that, despite the theoretical advantages of LSH-based attention, meaningful computation gains require sequence lengths substantially longer than those produced by typical high-resolution images.