Aliasing in Convnets: A Frame-Theoretic Perspective

TL;DR

This paper analyzes aliasing in convolutional neural networks using a frame-theoretic approach, providing stability bounds, optimization methods to suppress aliasing, and insights into initial random kernels.

Contribution

It introduces a novel frame-theoretic analysis of aliasing in convnets, deriving stability bounds, efficient optimization objectives, and statistical insights at initialization.

Findings

Frame-theoretic description of aliasing in convnets.

Efficient optimization objectives for promoting Parseval stability.

Closed-form expressions for aliasing effects at initialization.

Abstract

Using a stride in a convolutional layer inherently introduces aliasing, which has implications for numerical stability and statistical generalization. While techniques such as the parametrizations via paraunitary systems have been used to promote orthogonal convolution and thus ensure Parseval stability, a general analysis of aliasing and its effects on the stability has not been done in this context. In this article, we adapt a frame-theoretic approach to describe aliasing in convolutional layers with 1D kernels, leading to practical estimates for stability bounds and characterizations of Parseval stability, that are tailored to take short kernel sizes into account. From this, we derive two computationally very efficient optimization objectives that promote Parseval stability via systematically suppressing aliasing. Finally, for layers with random kernels, we derive closed-form expressions for the expected value and variance of the terms that describe the aliasing effects, revealing fundamental insights into the aliasing behavior at initialization.