Malleable 2.5D Convolution: Learning Receptive Fields along the Depth-axis for RGB-D Scene Parsing

TL;DR

This paper introduces a learnable 2.5D convolution operator that dynamically adapts the receptive field along the depth-axis for improved RGB-D scene parsing, avoiding fixed hyperparameters.

Contribution

It proposes a novel, differentiable malleable 2.5D convolution that learns depth receptive fields during training, seamlessly integrating into existing CNNs.

Findings

Improves semantic segmentation accuracy on NYUDv2 and Cityscapes datasets.

Demonstrates better generalization compared to fixed receptive field methods.

Achieves state-of-the-art results in RGB-D scene parsing.

Abstract

Depth data provide geometric information that can bring progress in RGB-D scene parsing tasks. Several recent works propose RGB-D convolution operators that construct receptive fields along the depth-axis to handle 3D neighborhood relations between pixels. However, these methods pre-define depth receptive fields by hyperparameters, making them rely on parameter selection. In this paper, we propose a novel operator called malleable 2.5D convolution to learn the receptive field along the depth-axis. A malleable 2.5D convolution has one or more 2D convolution kernels. Our method assigns each pixel to one of the kernels or none of them according to their relative depth differences, and the assigning process is formulated as a differentiable form so that it can be learnt by gradient descent. The proposed operator runs on standard 2D feature maps and can be seamlessly incorporated into pre-trained CNNs. We conduct extensive experiments on two challenging RGB-D semantic segmentation dataset NYUDv2 and Cityscapes to validate the effectiveness and the generalization ability of our method.