Pixel-Adaptive Convolutional Neural Networks

TL;DR

This paper introduces pixel-adaptive convolution (PAC), a modification of standard CNN convolutions that incorporates spatially-varying kernels based on local pixel features, improving performance and efficiency across various tasks.

Contribution

The paper proposes PAC, a flexible convolution variant that enhances CNNs by making filters spatially adaptive, and demonstrates its effectiveness in image upsampling, CRF approximation, and as a drop-in replacement.

Findings

PAC achieves state-of-the-art results in deep joint image upsampling.

PAC-CRF performs competitively with fully-connected CRFs but is faster.

Replacing standard convolutions with PAC improves performance in pre-trained networks.

Abstract

Convolutions are the fundamental building block of CNNs. The fact that their weights are spatially shared is one of the main reasons for their widespread use, but it also is a major limitation, as it makes convolutions content agnostic. We propose a pixel-adaptive convolution (PAC) operation, a simple yet effective modification of standard convolutions, in which the filter weights are multiplied with a spatially-varying kernel that depends on learnable, local pixel features. PAC is a generalization of several popular filtering techniques and thus can be used for a wide range of use cases. Specifically, we demonstrate state-of-the-art performance when PAC is used for deep joint image upsampling. PAC also offers an effective alternative to fully-connected CRF (Full-CRF), called PAC-CRF, which performs competitively, while being considerably faster. In addition, we also demonstrate that PAC can be used as a drop-in replacement for convolution layers in pre-trained networks, resulting in consistent performance improvements.