Processing Megapixel Images with Deep Attention-Sampling Models

TL;DR

This paper introduces an attention sampling model that enables processing of megapixel images efficiently by focusing computation on informative regions, reducing resource use while maintaining accuracy.

Contribution

It presents a novel differentiable attention sampling method that processes large images efficiently and can be trained end-to-end with standard SGD.

Findings

Reduces computation and memory by an order of magnitude.

Maintains accuracy comparable to classical models.

Focuses on informative image regions during sampling.

Abstract

Existing deep architectures cannot operate on very large signals such as megapixel images due to computational and memory constraints. To tackle this limitation, we propose a fully differentiable end-to-end trainable model that samples and processes only a fraction of the full resolution input image. The locations to process are sampled from an attention distribution computed from a low resolution view of the input. We refer to our method as attention sampling and it can process images of several megapixels with a standard single GPU setup. We show that sampling from the attention distribution results in an unbiased estimator of the full model with minimal variance, and we derive an unbiased estimator of the gradient that we use to train our model end-to-end with a normal SGD procedure. This new method is evaluated on three classification tasks, where we show that it allows to reduce computation and memory footprint by an order of magnitude for the same accuracy as classical architectures. We also show the consistency of the sampling that indeed focuses on informative parts of the input images.