Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

TL;DR

This paper introduces a neural network-based framework for fast, unsupervised scene understanding that explicitly reasons about objects by attending to scene elements sequentially, learning to infer object count, location, and class without supervision.

Contribution

It proposes a novel recurrent neural network approach that learns to perform efficient, unsupervised inference in structured image models, including variable-sized 2D and 3D scenes.

Findings

Successfully identifies multiple objects without supervision

Achieves accurate inference comparable to supervised models

Generalizes well to scenes with varying numbers of objects

Abstract

We present a framework for efficient inference in structured image models that explicitly reason about objects. We achieve this by performing probabilistic inference using a recurrent neural network that attends to scene elements and processes them one at a time. Crucially, the model itself learns to choose the appropriate number of inference steps. We use this scheme to learn to perform inference in partially specified 2D models (variable-sized variational auto-encoders) and fully specified 3D models (probabilistic renderers). We show that such models learn to identify multiple objects - counting, locating and classifying the elements of a scene - without any supervision, e.g., decomposing 3D images with various numbers of objects in a single forward pass of a neural network. We further show that the networks produce accurate inferences when compared to supervised counterparts, and that their structure leads to improved generalization.