2D-3D Geometric Fusion Network using Multi-Neighbourhood Graph Convolution for RGB-D Indoor Scene Classification

TL;DR

This paper introduces a novel 2D-3D fusion network that combines multi-neighbourhood graph convolution and voxel pooling to improve RGB-D indoor scene classification, outperforming current state-of-the-art methods.

Contribution

It proposes a new multi-neighbourhood graph convolution layer and an enhanced voxel pooling technique for robust 3D geometric feature extraction.

Findings

Outperforms state-of-the-art on NYU-Depth-V2 and SUN RGB-D datasets

Demonstrates improved accuracy in RGB-D indoor scene classification

Validates effectiveness of multi-neighbourhood graph convolution and Nearest Voxel Pooling

Abstract

Multi-modal fusion has been proved to help enhance the performance of scene classification tasks. This paper presents a 2D-3D Fusion stage that combines 3D Geometric Features with 2D Texture Features obtained by 2D Convolutional Neural Networks. To get a robust 3D Geometric embedding, a network that uses two novel layers is proposed. The first layer, Multi-Neighbourhood Graph Convolution, aims to learn a more robust geometric descriptor of the scene combining two different neighbourhoods: one in the Euclidean space and the other in the Feature space. The second proposed layer, Nearest Voxel Pooling, improves the performance of the well-known Voxel Pooling. Experimental results, using NYU-Depth-V2 and SUN RGB-D datasets, show that the proposed method outperforms the current state-of-the-art in RGB-D indoor scene classification task.