3D Gated Recurrent Fusion for Semantic Scene Completion

TL;DR

This paper introduces GRFNet, a novel 3D gated recurrent fusion network that adaptively combines RGB and depth data for improved semantic scene completion, demonstrating superior results on benchmark datasets.

Contribution

The paper presents a new multi-stage fusion strategy and a 3D gated recurrent network for effective multi-modal data fusion in SSC tasks.

Findings

GRFNet outperforms existing methods on benchmark datasets.

Multi-stage fusion models correlations among different stages.

Adaptive fusion improves semantic scene understanding.

Abstract

This paper tackles the problem of data fusion in the semantic scene completion (SSC) task, which can simultaneously deal with semantic labeling and scene completion. RGB images contain texture details of the object(s) which are vital for semantic scene understanding. Meanwhile, depth images capture geometric clues of high relevance for shape completion. Using both RGB and depth images can further boost the accuracy of SSC over employing one modality in isolation. We propose a 3D gated recurrent fusion network (GRFNet), which learns to adaptively select and fuse the relevant information from depth and RGB by making use of the gate and memory modules. Based on the single-stage fusion, we further propose a multi-stage fusion strategy, which could model the correlations among different stages within the network. Extensive experiments on two benchmark datasets demonstrate the superior performance and the effectiveness of the proposed GRFNet for data fusion in SSC. Code will be made available.