Convolutional Networks with MuxOut Layers as Multi-rate Systems for Image Upscaling

TL;DR

This paper interprets convolutional networks as adaptive filters combined with MuxOut layers for efficient image upscaling, providing a formal analysis of their filter effects and evaluating deterministic and probabilistic upscaling methods.

Contribution

It introduces a formal linear, space-variant model of convolutional networks with MuxOut layers and algorithms to analyze their filter effects for image upscaling.

Findings

Analyzed network filter effects at each location.

Evaluated deterministic upscalers for detail recovery.

Developed probabilistic upscalers that sample alias distributions.

Abstract

We interpret convolutional networks as adaptive filters and combine them with so-called MuxOut layers to efficiently upscale low resolution images. We formalize this interpretation by deriving a linear and space-variant structure of a convolutional network when its activations are fixed. We introduce general purpose algorithms to analyze a network and show its overall filter effect for each given location. We use this analysis to evaluate two types of image upscalers: deterministic upscalers that target the recovery of details from original content; and second, a new generation of upscalers that can sample the distribution of upscale aliases (images that share the same downscale version) that look like real content.