Dual PatchNorm
Manoj Kumar, Mostafa Dehghani, Neil Houlsby
TL;DR
This paper introduces Dual PatchNorm, a simple yet effective modification involving two LayerNorm layers in Vision Transformers, which consistently improves accuracy without negative effects.
Contribution
The paper presents a novel placement of LayerNorm layers in Vision Transformers, demonstrating its superiority over other configurations through extensive experiments.
Findings
Dual PatchNorm outperforms alternative LayerNorm placements.
Incorporating Dual PatchNorm improves accuracy of Vision Transformers.
The modification is trivial but consistently beneficial.
Abstract
We propose Dual PatchNorm: two Layer Normalization layers (LayerNorms), before and after the patch embedding layer in Vision Transformers. We demonstrate that Dual PatchNorm outperforms the result of exhaustive search for alternative LayerNorm placement strategies in the Transformer block itself. In our experiments, incorporating this trivial modification, often leads to improved accuracy over well-tuned Vision Transformers and never hurts.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCCD and CMOS Imaging Sensors · Advanced Memory and Neural Computing · Image Enhancement Techniques
MethodsAttention Is All You Need · Linear Layer · Residual Connection · Dense Connections · Multi-Head Attention · Position-Wise Feed-Forward Layer · Dropout · Softmax · Absolute Position Encodings · Byte Pair Encoding