Compress image to patches for Vision Transformer

TL;DR

This paper introduces CI2P-ViT, a hybrid CNN and Vision Transformer model that compresses images into patches to reduce computational costs and improve accuracy, demonstrating significant efficiency and performance gains.

Contribution

The paper presents a novel image compression-based patch generation method for ViT, reducing computational load and enhancing accuracy by integrating CNN inductive biases.

Findings

Achieved 92.37% accuracy on Animals-10, a 3.3% improvement over baseline.

Reduced FLOPs by 63.35%, significantly lowering computational costs.

Doubled training speed on identical hardware.

Abstract

The Vision Transformer (ViT) has made significant strides in the field of computer vision. However, as the depth of the model and the resolution of the input images increase, the computational cost associated with training and running ViT models has surged dramatically. This paper proposes a hybrid model based on CNN and Vision Transformer, named CI2P-ViT. The model incorporates a module called CI2P, which utilizes the CompressAI encoder to compress images and subsequently generates a sequence of patches through a series of convolutions. CI2P can replace the Patch Embedding component in the ViT model, enabling seamless integration into existing ViT models. Compared to ViT-B/16, CI2P-ViT has the number of patches input to the self-attention layer reduced to a quarter of the original. This design not only significantly reduces the computational cost of the ViT model but also effectively enhances the model's accuracy by introducing the inductive bias properties of CNN. The ViT model's precision is markedly enhanced. When trained from the ground up on the Animals-10 dataset, CI2P-ViT achieved an accuracy rate of 92.37%, representing a 3.3% improvement over the ViT-B/16 baseline. Additionally, the model's computational operations, measured in floating-point operations per second (FLOPs), were diminished by 63.35%, and it exhibited a 2-fold increase in training velocity on identical hardware configurations.