ViR:the Vision Reservoir

TL;DR

The paper introduces Vision Reservoir (ViR), a novel image classification method that replaces Transformer modules with a reservoir computing approach, reducing complexity and improving performance without pre-training.

Contribution

Proposes ViR, a reservoir computing-based alternative to ViT, addressing high computation and overfitting issues in image classification.

Findings

ViR outperforms ViT in accuracy without pre-training.

ViR has significantly fewer parameters and lower memory usage.

ViR demonstrates superior performance on multiple benchmarks.

Abstract

The most recent year has witnessed the success of applying the Vision Transformer (ViT) for image classification. However, there are still evidences indicating that ViT often suffers following two aspects, i) the high computation and the memory burden from applying the multiple Transformer layers for pre-training on a large-scale dataset, ii) the over-fitting when training on small datasets from scratch. To address these problems, a novel method, namely, Vision Reservoir computing (ViR), is proposed here for image classification, as a parallel to ViT. By splitting each image into a sequence of tokens with fixed length, the ViR constructs a pure reservoir with a nearly fully connected topology to replace the Transformer module in ViT. Two kinds of deep ViR models are subsequently proposed to enhance the network performance. Comparative experiments between the ViR and the ViT are carried out on several image classification benchmarks. Without any pre-training process, the ViR outperforms the ViT in terms of both model and computational complexity. Specifically, the number of parameters of the ViR is about 15% even 5% of the ViT, and the memory footprint is about 20% to 40% of the ViT. The superiority of the ViR performance is explained by Small-World characteristics, Lyapunov exponents, and memory capacity.