STR-GQN: Scene Representation and Rendering for Unknown Cameras Based on Spatial Transformation Routing

TL;DR

This paper introduces STR-GQN, a novel scene representation method that models spatial transformations without geometric priors, enhancing scene rendering for unknown camera views through a learnable routing mechanism.

Contribution

It proposes the Spatial Transformation Routing (STR) mechanism and an Occupancy Concept Mapping (OCM) framework, enabling better scene understanding without intrinsic camera data.

Findings

STR improves GQN performance on multiple datasets.

Routing passes observed information effectively between views.

Model demonstrates enhanced spatial cognition in scene rendering.

Abstract

Geometry-aware modules are widely applied in recent deep learning architectures for scene representation and rendering. However, these modules require intrinsic camera information that might not be obtained accurately. In this paper, we propose a Spatial Transformation Routing (STR) mechanism to model the spatial properties without applying any geometric prior. The STR mechanism treats the spatial transformation as the message passing process, and the relation between the view poses and the routing weights is modeled by an end-to-end trainable neural network. Besides, an Occupancy Concept Mapping (OCM) framework is proposed to provide explainable rationals for scene-fusion processes. We conducted experiments on several datasets and show that the proposed STR mechanism improves the performance of the Generative Query Network (GQN). The visualization results reveal that the routing process can pass the observed information from one location of some view to the associated location in the other view, which demonstrates the advantage of the proposed model in terms of spatial cognition.