DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs

TL;DR

This paper revitalizes DenseNets by refining training methods and architecture, demonstrating they can outperform modern models like Swin Transformer and ConvNeXt on ImageNet-1K and other tasks.

Contribution

It introduces improved training recipes and architectural adjustments that significantly enhance DenseNets' performance, surpassing recent state-of-the-art models.

Findings

DenseNets outperform Swin Transformer, ConvNeXt, and DeiT-III.

Models achieve near state-of-the-art results on ImageNet-1K.

Empirical analysis favors concatenation over additive shortcuts.

Abstract

This paper revives Densely Connected Convolutional Networks (DenseNets) and reveals the underrated effectiveness over predominant ResNet-style architectures. We believe DenseNets' potential was overlooked due to untouched training methods and traditional design elements not fully revealing their capabilities. Our pilot study shows dense connections through concatenation are strong, demonstrating that DenseNets can be revitalized to compete with modern architectures. We methodically refine suboptimal components - architectural adjustments, block redesign, and improved training recipes towards widening DenseNets and boosting memory efficiency while keeping concatenation shortcuts. Our models, employing simple architectural elements, ultimately surpass Swin Transformer, ConvNeXt, and DeiT-III - key architectures in the residual learning lineage. Furthermore, our models exhibit near state-of-the-art performance on ImageNet-1K, competing with the very recent models and downstream tasks, ADE20k semantic segmentation, and COCO object detection/instance segmentation. Finally, we provide empirical analyses that uncover the merits of the concatenation over additive shortcuts, steering a renewed preference towards DenseNet-style designs. Our code is available at https://github.com/naver-ai/rdnet.