Transformer in Transformer
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, Yunhe Wang

TL;DR
This paper introduces Transformer in Transformer (TNT), a novel visual transformer architecture that models attention within local patches to improve feature extraction across scales, achieving state-of-the-art accuracy on ImageNet.
Contribution
The paper proposes a new architecture, TNT, which incorporates attention within local patches to enhance visual feature representation in transformers.
Findings
Achieves 81.5% top-1 accuracy on ImageNet.
Outperforms similar models with comparable computational cost.
Demonstrates effectiveness of intra-patch attention in visual transformers.
Abstract
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 1616) as "visual sentences" and present to further divide them into smaller patches (e.g., 44) as "visual…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsAdvanced Neural Network Applications · Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques
MethodsLinear Layer · Absolute Position Encodings · Position-Wise Feed-Forward Layer · Adam · Layer Normalization · Byte Pair Encoding · Label Smoothing · Residual Connection · Transformer · Transformer in Transformer
