Adaptive Convolutions with Per-pixel Dynamic Filter Atom

TL;DR

This paper introduces a scalable, efficient method for adaptive convolutions with per-pixel dynamic filter atoms, enabling better modeling of intra-image variance without heavy computational costs.

Contribution

The paper proposes a novel filter decomposition approach using dynamic filter atoms generated by a lightweight network, supporting adaptive receptive fields efficiently.

Findings

Achieves comparable or better performance than traditional methods.

Effectively handles tasks with significant intra-image variance.

Maintains translation-equivariance and parameter efficiency.

Abstract

Applying feature dependent network weights have been proved to be effective in many fields. However, in practice, restricted by the enormous size of model parameters and memory footprints, scalable and versatile dynamic convolutions with per-pixel adapted filters are yet to be fully explored. In this paper, we address this challenge by decomposing filters, adapted to each spatial position, over dynamic filter atoms generated by a light-weight network from local features. Adaptive receptive fields can be supported by further representing each filter atom over sets of pre-fixed multi-scale bases. As plug-and-play replacements to convolutional layers, the introduced adaptive convolutions with per-pixel dynamic atoms enable explicit modeling of intra-image variance, while avoiding heavy computation, parameters, and memory cost. Our method preserves the appealing properties of conventional convolutions as being translation-equivariant and parametrically efficient. We present experiments to show that, the proposed method delivers comparable or even better performance across tasks, and are particularly effective on handling tasks with significant intra-image variance.