Nested-TNT: Hierarchical Vision Transformers with Multi-Scale Feature Processing

TL;DR

Nested-TNT introduces a hierarchical vision transformer that processes multi-scale features through nested algorithms, leading to improved image classification performance over existing models like ViT and TNT.

Contribution

The paper proposes a novel nested hierarchical transformer architecture that enhances multi-scale feature processing for vision tasks, surpassing prior models in accuracy.

Findings

Outperforms ViT and TNT on CIFAR10 and FLOWERS102 datasets.

Achieves over 2% improvement in classification accuracy.

Demonstrates better feature utilization through nested multi-scale processing.

Abstract

Transformer has been applied in the field of computer vision due to its excellent performance in natural language processing, surpassing traditional convolutional neural networks and achieving new state-of-the-art. ViT divides an image into several local patches, known as "visual sentences". However, the information contained in the image is vast and complex, and focusing only on the features at the "visual sentence" level is not enough. The features between local patches should also be taken into consideration. In order to achieve further improvement, the TNT model is proposed, whose algorithm further divides the image into smaller patches, namely "visual words," achieving more accurate results. The core of Transformer is the Multi-Head Attention mechanism, and traditional attention mechanisms ignore interactions across different attention heads. In order to reduce redundancy and improve utilization, we introduce the nested algorithm and apply the Nested-TNT to image classification tasks. The experiment confirms that the proposed model has achieved better classification performance over ViT and TNT, exceeding 2.25%, 1.1% on dataset CIFAR10 and 2.78%, 0.25% on dataset FLOWERS102 respectively.