Normalize Filters! Classical Wisdom for Deep Vision

TL;DR

This paper introduces a filter normalization technique inspired by classical image filters to improve deep vision models' robustness and generalization, especially under atmospheric transfer conditions.

Contribution

It proposes a simple normalization method for learned filters in deep networks, enhancing their atmosphere-equivariance and robustness, integrating classical filtering principles into modern deep learning.

Findings

Significant performance improvements on intensity variation benchmarks

ResNet34 outperforms CLIP with the proposed normalization

Normalized filters enhance robustness and generalization in deep vision models

Abstract

Classical image filters, such as those for averaging or differencing, are carefully normalized to ensure consistency, interpretability, and to avoid artifacts like intensity shifts, halos, or ringing. In contrast, convolutional filters learned end-to-end in deep networks lack such constraints. Although they may resemble wavelets and blob/edge detectors, they are not normalized in the same or any way. Consequently, when images undergo atmospheric transfer, their responses become distorted, leading to incorrect outcomes. We address this limitation by proposing filter normalization, followed by learnable scaling and shifting, akin to batch normalization. This simple yet effective modification ensures that the filters are atmosphere-equivariant, enabling co-domain symmetry. By integrating classical filtering principles into deep learning (applicable to both convolutional neural networks and convolution-dependent vision transformers), our method achieves significant improvements on artificial and natural intensity variation benchmarks. Our ResNet34 could even outperform CLIP by a large margin. Our analysis reveals that unnormalized filters degrade performance, whereas filter normalization regularizes learning, promotes diversity, and improves robustness and generalization.