Content-Adaptive Downsampling in Convolutional Neural Networks

TL;DR

This paper introduces an adaptive downsampling method for CNNs that selectively processes important image regions at higher resolutions, improving the balance between computational cost and accuracy.

Contribution

It proposes a novel adaptive downsampling scheme that dynamically allocates resolution based on regional importance, enhancing CNN efficiency.

Findings

Improves cost-accuracy trade-off in CNNs

Enhances fine detail recovery in dense prediction tasks

Versatile across different CNN architectures

Abstract

Many convolutional neural networks (CNNs) rely on progressive downsampling of their feature maps to increase the network's receptive field and decrease computational cost. However, this comes at the price of losing granularity in the feature maps, limiting the ability to correctly understand images or recover fine detail in dense prediction tasks. To address this, common practice is to replace the last few downsampling operations in a CNN with dilated convolutions, allowing to retain the feature map resolution without reducing the receptive field, albeit increasing the computational cost. This allows to trade off predictive performance against cost, depending on the output feature resolution. By either regularly downsampling or not downsampling the entire feature map, existing work implicitly treats all regions of the input image and subsequent feature maps as equally important, which generally does not hold. We propose an adaptive downsampling scheme that generalizes the above idea by allowing to process informative regions at a higher resolution than less informative ones. In a variety of experiments, we demonstrate the versatility of our adaptive downsampling strategy and empirically show that it improves the cost-accuracy trade-off of various established CNNs.