Patch Is Not All You Need
Changzhen Li, Jie Zhang, Yang Wei, Zhilong Ji, Jinfeng Bai, Shiguang, Shan

TL;DR
This paper introduces Patternformer, a novel approach that uses CNN-extracted patterns as input tokens for Transformers, preserving image structure and achieving state-of-the-art results on CIFAR datasets and competitive performance on ImageNet.
Contribution
The paper proposes Patternformer, which adaptively converts images into pattern sequences using CNNs, improving structural preservation and performance over traditional patch-based methods.
Findings
State-of-the-art on CIFAR-10 and CIFAR-100
Competitive results on ImageNet
Effective preservation of image structure
Abstract
Vision Transformers have achieved great success in computer visions, delivering exceptional performance across various tasks. However, their inherent reliance on sequential input enforces the manual partitioning of images into patch sequences, which disrupts the image's inherent structural and semantic continuity. To handle this, we propose a novel Pattern Transformer (Patternformer) to adaptively convert images to pattern sequences for Transformer input. Specifically, we employ the Convolutional Neural Network to extract various patterns from the input image, with each channel representing a unique pattern that is fed into the succeeding Transformer as a visual token. By enabling the network to optimize these patterns, each pattern concentrates on its local region of interest, thereby preserving its intrinsic structural and semantic information. Only employing the vanilla ResNet and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Image Processing Techniques and Applications · Medical Image Segmentation Techniques
Methods*Communicated@Fast*How Do I Communicate to Expedia? · Multi-Head Attention · Attention Is All You Need · Linear Layer · Position-Wise Feed-Forward Layer · Byte Pair Encoding · Adam · Label Smoothing · Kaiming Initialization · 1x1 Convolution
