Project-and-Fuse: Improving RGB-D Semantic Segmentation via Graph Convolution Networks

TL;DR

This paper introduces a novel RGB-D semantic segmentation method that uses late feature fusion guided by texture priors and employs Graph Neural Networks to improve segmentation quality, addressing misalignment and patch irregularity issues.

Contribution

It proposes a late fusion strategy with texture-guided feature injection and GNN-based patch relationship inference, along with improved 3D feature encoding and pixel feature preservation techniques.

Findings

Consistently outperforms existing methods on NYU-DepthV2 and SUN RGB-D datasets.

Effectively reduces segmentation irregular patches and misalignment issues.

Enhances feature extraction efficiency for depth maps.

Abstract

Most existing RGB-D semantic segmentation methods focus on the feature level fusion, including complex cross-modality and cross-scale fusion modules. However, these methods may cause misalignment problem in the feature fusion process and counter-intuitive patches in the segmentation results. Inspired by the popular pixel-node-pixel pipeline, we propose to 1) fuse features from two modalities in a late fusion style, during which the geometric feature injection is guided by texture feature prior; 2) employ Graph Neural Networks (GNNs) on the fused feature to alleviate the emergence of irregular patches by inferring patch relationship. At the 3D feature extraction stage, we argue that traditional CNNs are not efficient enough for depth maps. So, we encode depth map into normal map, after which CNNs can easily extract object surface tendencies.At projection matrix generation stage, we find the existence of Biased-Assignment and Ambiguous-Locality issues in the original pipeline. Therefore, we propose to 1) adopt the Kullback-Leibler Loss to ensure no missing important pixel features, which can be viewed as hard pixel mining process; 2) connect regions that are close to each other in the Euclidean space as well as in the semantic space with larger edge weights so that location informations can been considered. Extensive experiments on two public datasets, NYU-DepthV2 and SUN RGB-D, have shown that our approach can consistently boost the performance of RGB-D semantic segmentation task.