Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize

TL;DR

This paper introduces a new initialization method for sub-pixel convolution called convolution NN resize, which eliminates checkerboard artifacts and improves model performance compared to traditional resize convolution.

Contribution

The paper proposes a novel initialization technique for sub-pixel convolution that prevents checkerboard artifacts and enhances modeling power without increasing computational complexity.

Findings

Convolution NN resize is free from checkerboard artifacts immediately after initialization.

It has greater modeling power than resize convolution at the same computational cost.

It converges to solutions with smaller test errors.

Abstract

The most prominent problem associated with the deconvolution layer is the presence of checkerboard artifacts in output images and dense labels. To combat this problem, smoothness constraints, post processing and different architecture designs have been proposed. Odena et al. highlight three sources of checkerboard artifacts: deconvolution overlap, random initialization and loss functions. In this note, we proposed an initialization method for sub-pixel convolution known as convolution NN resize. Compared to sub-pixel convolution initialized with schemes designed for standard convolution kernels, it is free from checkerboard artifacts immediately after initialization. Compared to resize convolution, at the same computational complexity, it has more modelling power and converges to solutions with smaller test errors.