Fourier Sensitivity and Regularization of Computer Vision Models

TL;DR

This paper investigates the Fourier sensitivity of deep neural networks in computer vision, introduces a basis trick for measuring this sensitivity, and proposes a regularization method to improve model robustness by modifying frequency biases.

Contribution

It presents a novel basis trick for analyzing Fourier sensitivity and introduces a regularization framework to enhance robustness by controlling frequency biases in neural networks.

Findings

Models are sensitive to dataset-dependent frequencies.

Fourier-regularization improves robustness and accuracy.

Frequency bias can be effectively modified using the proposed regularizer.

Abstract

Recent work has empirically shown that deep neural networks latch on to the Fourier statistics of training data and show increased sensitivity to Fourier-basis directions in the input. Understanding and modifying this Fourier-sensitivity of computer vision models may help improve their robustness. Hence, in this paper we study the frequency sensitivity characteristics of deep neural networks using a principled approach. We first propose a basis trick, proving that unitary transformations of the input-gradient of a function can be used to compute its gradient in the basis induced by the transformation. Using this result, we propose a general measure of any differentiable model's Fourier-sensitivity using the unitary Fourier-transform of its input-gradient. When applied to deep neural networks, we find that computer vision models are consistently sensitive to particular frequencies dependent on the dataset, training method and architecture. Based on this measure, we further propose a Fourier-regularization framework to modify the Fourier-sensitivities and frequency bias of models. Using our proposed regularizer-family, we demonstrate that deep neural networks obtain improved classification accuracy on robustness evaluations.