# Learning Spatial Common Sense with Geometry-Aware Recurrent Networks

## TL;DR

This paper introduces geometry-aware recurrent networks that integrate 2D visual features into 3D scene representations, enabling improved scene understanding, novel view prediction, and persistent object detection in dynamic environments.

## Contribution

The paper presents a novel recurrent architecture with differentiable geometric operations for 3D scene understanding and object detection from visual sequences.

## Key findings

- Outperforms previous methods in novel view prediction
- Achieves persistent object detection over time
- Generalizes well to scenes with new objects and configurations

## Abstract

We integrate two powerful ideas, geometry and deep visual representation learning, into recurrent network architectures for mobile visual scene understanding. The proposed networks learn to "lift" and integrate 2D visual features over time into latent 3D feature maps of the scene. They are equipped with differentiable geometric operations, such as projection, unprojection, egomotion estimation and stabilization, in order to compute a geometrically-consistent mapping between the world scene and their 3D latent feature state. We train the proposed architectures to predict novel camera views given short frame sequences as input. Their predictions strongly generalize to scenes with a novel number of objects, appearances and configurations; they greatly outperform previous works that do not consider egomotion stabilization or a space-aware latent feature state. We train the proposed architectures to detect and segment objects in 3D using the latent 3D feature map as input--as opposed to per frame features. The resulting object detections persist over time: they continue to exist even when an object gets occluded or leaves the field of view. Our experiments suggest the proposed space-aware latent feature memory and egomotion-stabilized convolutions are essential architectural choices for spatial common sense to emerge in artificial embodied visual agents.