Depthwise Separable Convolutions Allow for Fast and Memory-Efficient Spectral Normalization

TL;DR

This paper introduces a simple, efficient method for spectral normalization of depthwise separable convolutions, significantly reducing computational and memory costs while maintaining effectiveness in image classification models like MobileNetV2.

Contribution

It presents a novel, low-overhead spectral normalization technique specifically designed for depthwise separable convolutions in neural networks.

Findings

Effective spectral normalization with negligible overhead

Improved control of spectral norms in lightweight models

Maintains accuracy in image classification tasks

Abstract

An increasing number of models require the control of the spectral norm of convolutional layers of a neural network. While there is an abundance of methods for estimating and enforcing upper bounds on those during training, they are typically costly in either memory or time. In this work, we introduce a very simple method for spectral normalization of depthwise separable convolutions, which introduces negligible computational and memory overhead. We demonstrate the effectiveness of our method on image classification tasks using standard architectures like MobileNetV2.