Pixel-wise Attentional Gating for Parsimonious Pixel Labeling

TL;DR

This paper introduces Pixel-wise Attentional Gating (PAG), a versatile mechanism that enables deep networks to selectively process spatial locations, improving efficiency and performance in various pixel labeling tasks without significant computational overhead.

Contribution

The paper presents PAG, a generic, architecture-independent gating mechanism that learns to dynamically allocate computation across spatial locations in deep networks for pixel labeling.

Findings

PAG achieves state-of-the-art or competitive results on multiple pixel labeling tasks.

PAG reduces computation by approximately 10% with minimal accuracy loss.

Dynamic spatial allocation improves performance trade-offs over static or layer-skipping methods.

Abstract

To achieve parsimonious inference in per-pixel labeling tasks with a limited computational budget, we propose a \emph{Pixel-wise Attentional Gating} unit (\emph{PAG}) that learns to selectively process a subset of spatial locations at each layer of a deep convolutional network. PAG is a generic, architecture-independent, problem-agnostic mechanism that can be readily "plugged in" to an existing model with fine-tuning. We utilize PAG in two ways: 1) learning spatially varying pooling fields that improve model performance without the extra computation cost associated with multi-scale pooling, and 2) learning a dynamic computation policy for each pixel to decrease total computation while maintaining accuracy. We extensively evaluate PAG on a variety of per-pixel labeling tasks, including semantic segmentation, boundary detection, monocular depth and surface normal estimation. We demonstrate that PAG allows competitive or state-of-the-art performance on these tasks. Our experiments show that PAG learns dynamic spatial allocation of computation over the input image which provides better performance trade-offs compared to related approaches (e.g., truncating deep models or dynamically skipping whole layers). Generally, we observe PAG can reduce computation by $10\%$ without noticeable loss in accuracy and performance degrades gracefully when imposing stronger computational constraints.