Depth CNNs for RGB-D scene recognition: learning from scratch better than transferring from RGB-CNNs

TL;DR

This paper introduces a novel bottom-up training strategy for depth CNNs in RGB-D scene recognition, emphasizing learning depth-specific features from limited data without relying on transfer learning from RGB models.

Contribution

It proposes an alternative training approach focusing on bottom layers and a modified CNN architecture, achieving state-of-the-art results without using pretrained RGB-CNNs.

Findings

Achieves state-of-the-art accuracy on NYU2 and SUN RGB-D datasets.

Demonstrates effective depth feature learning from limited data.

Outperforms transfer learning approaches in depth-specific feature extraction.

Abstract

Scene recognition with RGB images has been extensively studied and has reached very remarkable recognition levels, thanks to convolutional neural networks (CNN) and large scene datasets. In contrast, current RGB-D scene data is much more limited, so often leverages RGB large datasets, by transferring pretrained RGB CNN models and fine-tuning with the target RGB-D dataset. However, we show that this approach has the limitation of hardly reaching bottom layers, which is key to learn modality-specific features. In contrast, we focus on the bottom layers, and propose an alternative strategy to learn depth features combining local weakly supervised training from patches followed by global fine tuning with images. This strategy is capable of learning very discriminative depth-specific features with limited depth images, without resorting to Places-CNN. In addition we propose a modified CNN architecture to further match the complexity of the model and the amount of data available. For RGB-D scene recognition, depth and RGB features are combined by projecting them in a common space and further leaning a multilayer classifier, which is jointly optimized in an end-to-end network. Our framework achieves state-of-the-art accuracy on NYU2 and SUN RGB-D in both depth only and combined RGB-D data.