Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures

TL;DR

This paper introduces SimpleNet, a straightforward 13-layer CNN architecture that achieves competitive or superior accuracy compared to deeper, more complex models while significantly reducing computational and memory requirements, making it suitable for resource-limited systems.

Contribution

The paper presents a simple, principled design for CNNs that outperforms complex architectures like VGGNet and ResNet on multiple benchmarks with fewer parameters.

Findings

SimpleNet outperforms VGGNet, ResNet, and GoogleNet on several benchmarks.

Achieves state-of-the-art on CIFAR10 with fewer parameters.

Performs competitively on ImageNet with a simpler architecture.

Abstract

Major winning Convolutional Neural Networks (CNNs), such as AlexNet, VGGNet, ResNet, GoogleNet, include tens to hundreds of millions of parameters, which impose considerable computation and memory overhead. This limits their practical use for training, optimization and memory efficiency. On the contrary, light-weight architectures, being proposed to address this issue, mainly suffer from low accuracy. These inefficiencies mostly stem from following an ad hoc procedure. We propose a simple architecture, called SimpleNet, based on a set of designing principles, with which we empirically show, a well-crafted yet simple and reasonably deep architecture can perform on par with deeper and more complex architectures. SimpleNet provides a good tradeoff between the computation/memory efficiency and the accuracy. Our simple 13-layer architecture outperforms most of the deeper and complex architectures to date such as VGGNet, ResNet, and GoogleNet on several well-known benchmarks while having 2 to 25 times fewer number of parameters and operations. This makes it very handy for embedded systems or systems with computational and memory limitations. We achieved state-of-the-art result on CIFAR10 outperforming several heavier architectures, near state of the art on MNIST and competitive results on CIFAR100 and SVHN. We also outperformed the much larger and deeper architectures such as VGGNet and popular variants of ResNets among others on the ImageNet dataset. Models are made available at: https://github.com/Coderx7/SimpleNet