Deep Isometric Learning for Visual Recognition

TL;DR

This paper introduces deep isometric convolutional networks that can be trained effectively without normalization or skip connections by maintaining near-isometric properties in kernels and activations, achieving competitive performance.

Contribution

It demonstrates that deep vanilla ConvNets can be trained without normalization or skip connections using near-isometric initialization and activations, challenging conventional training techniques.

Findings

Near-isometric networks perform comparably to ResNet on ImageNet.

Near-isometric networks outperform ResNet on COCO.

Training without normalization is feasible with isometric constraints.

Abstract

Initialization, normalization, and skip connections are believed to be three indispensable techniques for training very deep convolutional neural networks and obtaining state-of-the-art performance. This paper shows that deep vanilla ConvNets without normalization nor skip connections can also be trained to achieve surprisingly good performance on standard image recognition benchmarks. This is achieved by enforcing the convolution kernels to be near isometric during initialization and training, as well as by using a variant of ReLU that is shifted towards being isometric. Further experiments show that if combined with skip connections, such near isometric networks can achieve performances on par with (for ImageNet) and better than (for COCO) the standard ResNet, even without normalization at all. Our code is available at https://github.com/HaozhiQi/ISONet.