Depth-Adapted CNN for RGB-D cameras

TL;DR

This paper introduces a depth-adapted CNN that integrates geometric information from RGB-D cameras by using depth as a spatial offset, enhancing feature extraction while maintaining invariance to scale and rotation.

Contribution

It proposes a novel CNN architecture that incorporates depth as a spatial offset, improving geometric feature extraction over traditional RGB CNNs.

Findings

The model is invariant to scale and rotation around X and Y axes.

It reduces to a regular CNN when depth data is constant.

Experimental results validate the effectiveness of the approach.

Abstract

Conventional 2D Convolutional Neural Networks (CNN) extract features from an input image by applying linear filters. These filters compute the spatial coherence by weighting the photometric information on a fixed neighborhood without taking into account the geometric information. We tackle the problem of improving the classical RGB CNN methods by using the depth information provided by the RGB-D cameras. State-of-the-art approaches use depth as an additional channel or image (HHA) or pass from 2D CNN to 3D CNN. This paper proposes a novel and generic procedure to articulate both photometric and geometric information in CNN architecture. The depth data is represented as a 2D offset to adapt spatial sampling locations. The new model presented is invariant to scale and rotation around the X and the Y axis of the camera coordinate system. Moreover, when depth data is constant, our model is equivalent to a regular CNN. Experiments of benchmarks validate the effectiveness of our model.