Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference

TL;DR

This paper introduces a dynamic convolution method that selectively processes important image regions, leading to faster inference and improved focus on relevant areas without sacrificing accuracy.

Contribution

We propose a novel dynamic convolution technique with input-dependent gating, trained end-to-end, that enhances efficiency and accuracy in CNNs, especially for spatially sparse tasks.

Findings

Achieves 60% faster inference on human pose estimation without accuracy loss.

Outperforms existing methods in accuracy and computational efficiency on CIFAR, ImageNet, and MPII.

Provides an efficient CUDA implementation for real-world deployment.

Abstract

Modern convolutional neural networks apply the same operations on every pixel in an image. However, not all image regions are equally important. To address this inefficiency, we propose a method to dynamically apply convolutions conditioned on the input image. We introduce a residual block where a small gating branch learns which spatial positions should be evaluated. These discrete gating decisions are trained end-to-end using the Gumbel-Softmax trick, in combination with a sparsity criterion. Our experiments on CIFAR, ImageNet and MPII show that our method has better focus on the region of interest and better accuracy than existing methods, at a lower computational complexity. Moreover, we provide an efficient CUDA implementation of our dynamic convolutions using a gather-scatter approach, achieving a significant improvement in inference speed with MobileNetV2 residual blocks. On human pose estimation, a task that is inherently spatially sparse, the processing speed is increased by 60% with no loss in accuracy.