Spectral Wavelet Dropout: Regularization in the Wavelet Domain

TL;DR

Spectral Wavelet Dropout (SWD) is a new regularization method for CNNs that drops frequency bands in wavelet decompositions, improving generalization with fewer hyperparameters and lower computational cost compared to Fourier-based methods.

Contribution

This work introduces SWD, a novel wavelet domain dropout technique with two variants, and compares it to existing Fourier-based spectral dropout methods, demonstrating improved efficiency and competitive performance.

Findings

SWD variants outperform Fourier Dropout in benchmarks.

SWD requires only one hyperparameter, simplifying tuning.

SWD has lower computational complexity during training.

Abstract

Regularization techniques help prevent overfitting and therefore improve the ability of convolutional neural networks (CNNs) to generalize. One reason for overfitting is the complex co-adaptations among different parts of the network, which make the CNN dependent on their joint response rather than encouraging each part to learn a useful feature representation independently. Frequency domain manipulation is a powerful strategy for modifying data that has temporal and spatial coherence by utilizing frequency decomposition. This work introduces Spectral Wavelet Dropout (SWD), a novel regularization method that includes two variants: 1D-SWD and 2D-SWD. These variants improve CNN generalization by randomly dropping detailed frequency bands in the discrete wavelet decomposition of feature maps. Our approach distinguishes itself from the pre-existing Spectral "Fourier" Dropout (2D-SFD), which eliminates coefficients in the Fourier domain. Notably, SWD requires only a single hyperparameter, unlike the two required by SFD. We also extend the literature by implementing a one-dimensional version of Spectral "Fourier" Dropout (1D-SFD), setting the stage for a comprehensive comparison. Our evaluation shows that both 1D and 2D SWD variants have competitive performance on CIFAR-10/100 benchmarks relative to both 1D-SFD and 2D-SFD. Specifically, 1D-SWD has a significantly lower computational complexity compared to 1D/2D-SFD. In the Pascal VOC Object Detection benchmark, SWD variants surpass 1D-SFD and 2D-SFD in performance and demonstrate lower computational complexity during training.