Learning Common Representation from RGB and Depth Images

TL;DR

This paper introduces a novel deep learning architecture that learns a shared representation from RGB and depth images, enabling improved semantic segmentation and depth prediction, including cross-modality tasks.

Contribution

It proposes a common deep representation for RGB-D data, replacing traditional feature fusion, allowing cross-modality inference and joint learning for segmentation and depth estimation.

Findings

Effective in semantic segmentation and depth prediction

Enables cross-modality tasks using a single modality at test time

Demonstrates superior performance on RGB-D datasets

Abstract

We propose a new deep learning architecture for the tasks of semantic segmentation and depth prediction from RGB-D images. We revise the state of art based on the RGB and depth feature fusion, where both modalities are assumed to be available at train and test time. We propose a new architecture where the feature fusion is replaced with a common deep representation. Combined with an encoder-decoder type of the network, the architecture can jointly learn models for semantic segmentation and depth estimation based on their common representation. This representation, inspired by multi-view learning, offers several important advantages, such as using one modality available at test time to reconstruct the missing modality. In the RGB-D case, this enables the cross-modality scenarios, such as using depth data for semantically segmentation and the RGB images for depth estimation. We demonstrate the effectiveness of the proposed network on two publicly available RGB-D datasets. The experimental results show that the proposed method works well in both semantic segmentation and depth estimation tasks.

Tables

Table 1. TABLE I: Independent and joint learning with one or two views at test time. The best results are shown in bold.

Setting	Branch	Independent		Common representation
		RGB	Depth	RGB	Depth	RGB+D
NYU2 dataset
SS	RGB-SS	53.1	-	54.1	41.2	57.6
	D-SS	-	37.1	54.2	41.1	57.7
SS-D	RGB-SS	53.1	-	54.3	35.0	55.2
	D-D	-	0.51	0.72	0.39	0.53
SUN dataset
SS	RGB-SS	39.7	-	39.4	31.1	42.4
	D-SS	-	31.1	39.4	31.1	42.3
SS-D	RGB-SS	39.7	-	39.3	20.3	39.9
	D-D	-	0.36	0.62	0.31	0.31

Table 2. TABLE II: Comparison to the state of art on different tasks.

Methods	Sem. Segmentation			Depth Est.
	RGB	D	RGB+D	RGB
NYU2 dataset
Our method	54.1	41.2	57.6	0.72
Li et al. [20]	-	-	-	0.82
Roy et al. [26]	-	-	-	0.74
Laina et al. [19]	-	-	-	0.57
Eigen et al. [6]	-	-	52.6	0.64
FuseNet-SF3 [13]	-	-	56.0	-
MVCNet-MaxPool [23]	-	-	59.0	-
SUN dataset
Our method	39.49	31.1	42.4	0.62
Segnet [1]	22.1	-	-	-
Bayesian-Segnet [17]	30.7	-	-	-
Hazirbas et Ma [13]	32.4	28.8	33.6	-
FuseNet-SF5 [13]	-	-	37.3	-
DFCN-DCRF [16]	-	-	39.3	-
Context-CRF [27]	42.3	-	-	-
RefineNet [22]	45.9	-	-	-
CFN [21]	-	-	48.1	-

Equations

$$L_{rgb}^{ss}=-\sum _{i=1}^{N}P({\mathbf{y}}_i|{\hat{\mathbf{x}}}_i),$$

$$D(x)=\left\{\begin{array}[]{ll}0.5x^{2},\quad\quad\textnormal{if}\,|x|<1\\ |x|-0.5,\quad{\rm otherwise.}\end{array}\right.$$

$$h(\mathbf{z})=h({\mathbf{W}}_x⊛{\mathbf{x}}^p+{\mathbf{W}}_d⊛{\mathbf{d}}^p+\mathbf{b}),$$

$${\mathbf{z}}^r=g([{\mathbf{V}}_x⊛h(\mathbf{z}),{\mathbf{V}}_d⊛h(\mathbf{z})]+{\mathbf{b}}_r),$$

$$\sum _{i=1}^N\{L_r[{\mathbf{z}}_i,g(h({\mathbf{z}}_i))]+L_r[{\mathbf{z}}_i,g(h({\mathbf{x}}_i^p))]+L_r[{\mathbf{z}}_i,g(h({\mathbf{d}}_i^p))]\},$$

$$\mathcal{L}=L_{rgb}^{ss}+L_d+\lambda L_{rec},$$

Full text

I INTRODUCTION

Visual scene understanding is a critical capability enabling robots to act in their working environment. Modern robots and autonomous vehicles are equipped with many, often complementary sensing technologies. Multiple sensors aim to satisfy the need for the redundancy and robustness critical for achieving the human level of the driving safety.

The most frequent case is RGB-D cameras collecting color and depth information for different computer vision tasks [5, 14, 28]. As information collected by the depth camera is complementary to RGB images, the depth can potentially help decode structural information of the scene and improve the performance on such tasks as object detection and semantic segmentation [28].

Development of convolutional neural networks (CNNs) boosted the performance of the image classification. Recently, their success has been replicated on object detection and semantic segmentation tasks. The key contribution of CNN models lies in their ability to model complex visual scenes. Current CNN-based approaches provide the state-of-the-art performance in semantic segmentation benchmarks [4, 9].

When RGB images are completed with depth information, the straightforward idea is to incorporate depth information into a semantic segmentation framework. Different methods have been developed including deep features pooling, dense feature, multi-scale fusion, etc. [7, 8, 11, 18, 24]. Most recent methods, like FuseNet [13, 15], use an encoder-decoder architecture, where the encoder part is composed of two branches that simultaneously extract features from RGB and depth images and fuse depth features into the RGB feature maps. Moreover, training individual RGB and depth views has been replaced with the joint learning. It was shown that the semantics predictions of jointly learned network can be fused more consistently than predictions of a network trained on individual views [23].

In this paper, we propose a new deep learning architecture for the tasks of semantic segmentation and depth estimation from RGB-D images (see Figure 1 for different scenarios). Usually, these tasks are addressed separately, with a special design for semantic segmentation [13, 23] or depth prediction [7]. We develop a unifying framework capable to cope with either task.

We adopt the multi-view approach to RGB-D data, where RGB and depth are complementary sources of information about a visual scene. All existing methods, whether they train the view models independently or jointly, adopt the fusion-based representation. The feature fusion takes benefit from the view complementarity to reduce the uncertainty of segmentation and labelling. The fusion-based approaches however require both views to be available at test time.

We revise the fusion-based approach and replace it with the common representation [25]. Adopting the principle of the common representation gives a number of benefits well studied in the multi-view learning [2]. First, it allows to obtain the common representation from one view and then reconstruct all other views. It can accomplish the task when one view is unavailable due to technical or other reasons, thus increasing the robustness and fault-tolerance of the system. Working with one-view data at test time enables cross-view scenarios rarely addressed in the state of art. In semantic segmentation, when the RGB view is unavailable, the depth view can be used to obtain the common representation and accomplish the semantic segmentation task. And vice-versa, in the case of a depth estimation, the common representation allows to use the RGB view to reconstruct the depth of a scene or an object.

Second, the common representation is the central component allowing to deploy the same architecture for both RGB-D tasks. Representation common to RGB and depth allows to enforce the consistency between the views and improve the segmentation quality and depth estimation accuracy.

Third, the proposed architecture encourages a higher modularity of the deep network. Our proposal combines the state of art components, the encoder-decoder networks for semantic segmentation and a multi-view autoencoder for the common representation. The system can then benefit from any progress in individual components. The modularity allows to upgrade a component without changing the entire system, training and optimization routines.

The remainder of the paper is organized as follows. In Section II, we review the state of art of semantic segmentation and depth estimation for RGB-D images. In Section III, we introduce the multi-view deep architecture and describe in details each component, the two-stage training and optimization. Section IV reports results of evaluating the network on two public RGB-D datasets; it also discusses some open questions. Section V concludes the paper.

II STATE OF ART

Depth representation. Depth information is rarely used in any segmentation network as raw data, most methods use so called HHA representation of the depth [11]. This representation consists of three channels: disparity, height of the pixels and the angle between normals and the gravity vector based on the estimated ground floor. The color code provided by HHA helps visualize depth information; it can reveal some patterns that resemble RGB patterns.

Semantic segmentation and depth estimation. These two fundamental tasks for RGB-D images are strongly correlated and mutually beneficial, and most efforts were on putting both views in one architecture. In particular, with the success of CNN architectures, many methods aimed to inject the depth information into the semantic segmentation network [8, 13, 18, 15, 23, 29].

Ladicky at al. [18] were first to replace single-view depth estimation and semantic segmentation by a joint training model. They considered both semantic label loss and depth label loss when learning a classifier. Using properties of perspective geometry, they reduced the learning of a pixel-wise depth classifier to a simpler classifier predicting one of fixed canonical depth values [18].

Two separate CNN processing streams, one for each modality, were proposed by Eitel at al. [8]; they are consecutively combined in a late fusion network. The method also introduced a multi-stage training methodology for handling depth data with CNNs. It used the HHA representation of depth and the data augmentation scheme for robust learning with depth image.

A unified framework for joint depth and semantic prediction was proposed by Wang at al. [30]. Given an image, they first use a trained CNN to jointly predict a global layout composed of pixel-wise depth values and semantic labels. The joint network showed to provide more accurate depth prediction than a state-of-the-art CNN trained solely for depth prediction. To further obtain fine-level details, the image is decomposed into local segments for region-level depth and semantic prediction.

By considering RGB and depth channels as multi-view data, [23] enforced the multi-view consistency during training and testing. At test time, the semantic predictions of the network are fused more consistently than predictions of a network trained on individual views. The network architecture uses a recent single-view deep learning approach to RGB and depth fusion and enhances it with multi-scale loss minimization.

FuseNet [13] developed an encoder-decoder type network, where the encoder part is composed of two branches of networks that simultaneously extract features from RGB and depth images and fuse depth features into the RGB feature maps as the network goes deeper.

Although most of the above methods apply the late fusion, it is also possible to fuse depth information into the early layers of fully convolutional neural network [15]. Coupled with the dilated convolution for later contextual reasoning, it combines a depth-sensitive fully-connected CRF with the previous convolution layers to refine the preliminary result.

Depth Completion. The problem of completing the depth channel of an RGB-D image has been addressed in [31]. Indeed, it often the case that commodity-grade depth cameras fail to sense depth for bright, transparent, and distant surfaces thus leaving entire holes in the depth images. They train a deep network that takes an RGB image as input and predicts dense surface normals and occlusion boundaries. Those predictions are then combined with raw depth observations provided by the RGB-D camera to solve for depths for all pixels, including those missing in the original observation.

II-A Multi-view learning

In the previous section we reviewed different ways to fuse RGB and depth features. Meanwhile, there exist alternative representations for multi-view data [2]. One such alternative, the common representation learning (CRL), tries to embed different views of the data in a common subspace [2]. It allows to obtain a common representation from one view and use it to reconstruct other views.

Two complementary approaches to CRL are based on canonical correlation analysis (CCA) and multi-modal autoencoders (MAE). CCA based approaches learn a joint representation by maximizing correlation of the views when projected into the common subspace. Second approach to embed two views is based on multi-modal autoencoders (MAEs) [25]. The idea is to train an autoencoder able to perform two kinds of reconstruction. Given one view, the model learns both self-reconstruction and cross-reconstruction (reconstruction of the other view).

As CCA and MAE based approaches appear to be complementary, several methods tried to combine them in one framework [30]. For example, a MAE based approach called Correlational Neural Network (CorrNet) [3] tried to explicitly maximize the correlation between the views when projecting them into the common subspace.

III DEEP ARCHITECTURE

We aim to solve two fundamental tasks for RGB-D images: semantic segmentation and depth prediction. We assume that we are given a training set of $N$ RGB-D images (${\bf x}_{i},{\bf d}_{i}$), $i=1,\ldots,N$. All images are assumed to be resized to width $W$ and height $H$. Depth images are in HHA representation and have the same value range as RGB images, ${\bf x}_{i},{\bf d}_{i}\in R^{H\times W\times 3}$. RGB images are annotated with ${\bf y}_{i}\in L^{H\times W}$, where $L$ is the label set, $L=\{1,\ldots,K\}$. In the case of depth estimation, we assume to have additionally the ground true values ${\bf d}^{*}_{i}\in R^{H\times W}$.

We propose an architecture composed of two separate branches, one for each modality, which are consecutively fed into a common representation network. Two individual modality networks are of the encoder-decoder type, where the encoder applies dilated convolution to extract an informative feature map, while the decoder applies ”atrous” convolution at multiple scales to encode contextual information and refine the segmentation boundaries. This choice is motivated by the recent success of the encoder-decoder architecture of DeepLab network [4]. It has been also used in FuseNet [13] and SegNet [1] and has showed good segmentation performance.

Both RGB and depth encoders are initialized by the Resnet101 model trained on the COCO dataset. The encoders generate the feature maps, which the decoders use in ”atrous” spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales.

Feature maps generated by two modality branches are fed to the common representation network implemented in the form of a multi-view autoecoder [30]. Unlike the conventional fusion of RGB and depth feature maps, the multi-view autoencoder allows to extract the shared representation from either one or two views.

III-A Training RGB network

Our architecture enables two different settings. In the semantic segmentation (SS) setting, both RGB and depth views are jointly used to segment an image. In the case of segmentation and depth estimation (SS-D), we face the multi-task setting and expect the two views to achieve good results in both tasks.

We first proceed by training two individual modality branches (see the purple and blue streams in Figure 2). Let $\theta^{I}$ and $\theta^{D}$ be parameters of RGB and depth networks, respectively. Let ${\bf x}^{p}_{i}=g^{I}({\bf x}_{i};\theta^{I})$ be the feature map extracted from the last layer of the RGB decoder when applied to image ${\bf x}_{i}$, ${\bf x}^{p}_{i}\in R^{d\times H^{{}^{\prime}}\times W^{{}^{\prime}}}$. Analogously, let ${\bf d}^{p}_{i}=g^{D}({\bf d}_{i};\theta^{D})$ be the feature map extracted from depth decoder when applied to the depth image ${\bf d}_{i}$, ${\bf d}^{p}_{i}\in R^{d\times H^{{}^{\prime}}\times W^{{}^{\prime}}}$.

The network is trained in two stages. We first train the modality branches, then we train the entire network. In the first stage, we place a randomly initialized softmax classification layer on top of $g^{I}$ and train the RGB network to minimize the semantic loss of the training data. The semantic loss $L_{rgb}^{ss}$ is defined as the cross-entropy loss

[TABLE]

where $\widehat{\bf x}_{i}$ is a pixel-wise prediction for image ${\bf x}_{i}$, ${\bf y}_{i}$ is the ground truth labels, $P({\bf y}|{\bf x})=\sum_{j}{\rm log}\ p(y_{j}|x_{j})$, and $p(y_{j}|x_{j})$ is the probability of semantic label $y_{j}$ at pixel $j$.

The network is trained using the stochastic gradient descent on mini-batches of RGB images. After the convergence, all parameters $\theta^{I}$ of the RGB network are kept for the second stage, except the last layer which will be replaced by the common representation and reconstruction layer.

III-B Training depth network

Training the depth branch depends on the task. In the SS setting, the depth network is trained, similarly to the RGB network, to minimize semantic loss $L_{d}^{ss}=-\sum^{N}_{i=1}P({\bf y}_{i}|\widehat{\bf d}_{i}),$ where $\hat{\bf d}_{i}$ is a prediction for depth image ${\bf d}_{i}$.

In the SS-D setting, we train the depth branch to minimize the regression loss on the training depth data. We tested several state-of-art proposals for the loss function $L_{d}^{d}$. One is the scale-invarient loss [7]; it measures the relationships between points in the image irrespectively of the absolute values. We also considered the standard $L_{2}$ and smoothed $L_{1}$ loss [10]. Less sensitive to outliers that the $L_{2}$ loss, smoothed $L_{1}$ loss is defined as $L^{smo}_{1}({\bf d}_{i}^{*},\widehat{\bf d}_{i})=\sum_{j}D({\bf d}_{ij}^{*}-\widehat{\bf d}_{ij}),$ where

[TABLE]

III-C Common representation

Common representation network is implemented as a multi-view autoencoder [3, 25]. It includes a hidden layer and an output layer. The input to the hidden layer is two feature maps ${\bf x}^{p},{\bf d}^{p}$ from two modality branches. Similar to conventional autoencoders, the input and output layer has the same shape as the input, $d\times H^{\prime}\times W^{\prime}$, whereas the hidden layer is shaped as $k\times H^{\prime}\times W^{\prime}$, with $k$ being often smaller than $d$ (in Figure 2, $d$=256 and $k$=128).

Given a two-view input ${\bf z}=({\bf x}^{p},{\bf d}^{p})$, the hidden layer computes an encoded representation as the convolution

[TABLE]

where ${\bf W}_{x}$, ${\bf W}_{d}$ are projection weights, ${\bf b}$ is a bias vector, and $h$ is an activation function, such as sigmoid or tanh.

The output layer tries to reconstruct ${\bf z}$ from this hidden representation $h({\bf z})$ by computing

[TABLE]

where ${\bf V}_{x}$, ${\bf V}_{d}$ are reconstruction weights, ${\bf b}_{r}$ is a output bias vector and $g$ is an activation function.

Given data $\{({\bf x}^{p}_{i},{\bf d}^{p}_{i})\}_{i=1}^{N}$ from RGB and depth branches, the common representation is designed to minimize the self- and cross-reconstruction errors. The first is the of reconstructing ${\bf x}^{r}_{i}$ from ${\bf x}^{p}_{i}$ and ${\bf d}^{r}_{i}$ from ${\bf d}^{p}_{i}$. The second one is the error of reconstructing ${\bf x}^{r}_{i}$ from ${\bf d}^{p}_{i}$ and ${\bf d}^{r}_{i}$ from ${\bf x}^{p}_{i}$.

To achieve this goal, we try to find the parameter values ${\theta^{A}}=\{{\bf W}_{x},{\bf W}_{d},{\bf V}_{x},{\bf V}_{d},{\bf b},{\bf b}_{r}\}$ by minimizing the reconstruction loss function $L_{rec}$, defined as follows

[TABLE]

where $L_{r}$ is the reconstruction error, $L_{r}({\bf x},{\bf x}^{\prime})=||{\bf x}-{\bf x}^{\prime}||^{2}$.

III-D End-to-end learning

Common representation allows to obtain the reconstructed feature maps for both RGB and depth images as $f(g({\bf x}^{p}))$ and $f(g({\bf d}^{p}))$. The entire set of network parameters is $\theta=\{\theta^{I},\theta^{D},\theta^{A}\}$. The objective function to minimize is then defined as

[TABLE]

where the depth branch loss $L_{d}$ is $L^{ss}_{d}$ in the SS setting and $L^{d}_{d}$ in the SS-D setting; $\lambda$ is a scaling parameter for the reconstruction loss. In the above formulation, the semantic, depth and reconstruction losses are optimized jointly.

In addition to the common representation and view reconstruction, we also considered a possibility of maximizing the view correlation, as suggested in CorrNet [3]. In such a case, we try to maximize the correlation between the hidden representations of the two views. The correlation term can be included in the objective function $\cal L$, it makes sure that the hidden representations of the two views are highly correlated.

III-E Training and optimization

The architecture is implemented on the PyTorch framework. At the first stage, we train individual branches independently. In the SS setting, we train RGB and depth branches with segmentation labels, they are denoted RGB-SS and D-SS. Each branch is trained for 20,000 iterations using SGD with momentum 0.9, batch size 24, and minimizing modality losses, $L^{ss}_{rgb}$ and $L_{d}^{ss}$. We retain the model parameters $\theta^{I}$ and $\theta^{D}$ for the second stage.

In the SS-D setting, we train the RGB branch with segmentation labels (RGB-SS) and the depth branch with depth ground truth (D-D). The D-D branch is trained using the scale irrelevant loss $L_{2}$ loss or the smooth $L_{1}$ loss. We apply weight decay=0.0005 and the polynomial decay for the learning rate, with the base LR=0.0001 and power=0.9.

In the second stage, we train the entire network. We start with the two branch parameters $\theta^{I}$ and $\theta^{D}$ trained in the first stage, and refine them as well as the common representation network $\theta^{A}$ by minimizing the objective function $\cal L$ which combines semantic, depth and reconstruction losses. We fine-tune the network end-to-end with the Adam optimizer, but we freeze parameters of two modality encoders, it allows to speed-up the training without performance loss.

For the segmentation task, we add data augmentation, by flipping and randomly rotating input images on an angle between [-10, 10] degrees. RGB-D images to be augmented are selected randomly, but the augmentation is identical for both views.

IV EVALUATION

We evaluated the proposed network on two publicly available RGB-D datasets: NYU depth dataset, 2nd version [24] and SUN [28]. NYU2 is a popular dataset, with 27 indoor categories but not all well represented. Following the publicly available split [24], 27 categories are reorganized, including 13 most common categories and an ’other’ category for all remaining images. The training/test split is 795/654 images. Images are resized to $512\times 512$ at training time, full size images are used at test time. SUN dataset contains 10,335 RGB-D images with 40 categories [28]. Following the publicly available split with 37 most common and other categories [12], it consists of 5,285 images for training and 5,050 images for testing. Images are resized to $360\times 360$ at training time; full size images are used at test time. All depth images are encoded using the HHA representation.

IV-A Qualitative analysis

We start with the qualitative analysis and test the proposed architecture with exemplar RGB-D images. Figure 3.a shows how a NYU2 example gets processed by the network. In addition to the input images and ground truth segmentation, it shows feature maps extracted at different layers of the network. The upper row refers to the RGB branch, the lower row refers to depth branch. Column 2 visualizes feature maps generated by two modality decoders. Column 3 shows the common representations obtained from each modality map. A close resemblance of the two maps supports the concept of common representation which can be obtained from either view. Then, the reconstructed feature maps for both views are shown in column 4 and final predictions in column 5.

Figure 3.b shows the cross-view reconstruction, where the RGB image is only available at test time. It starts with feature maps extracted from RGB network and the common representation. Then it shows how the common representation is used for two reconstruction and prediction maps.

IV-B Quantitative results

Modularity. The proposed architecture is designed in a modular way. It does not use any particular techniques invented to improve the semantic segmentation in special cases and image regions, such as multi-scale input, CRF, overlapping windows and cross-view ambiguities [22, 15]. This choice is motivated by two main reasons. First, we wanted to test the effectiveness of the CRL in isolation, by excluding any impact of the additional improvements and the comparing to the state of art baseline. We paid most attention to the multi-view autoencoder and its capacity to generate common representation and reconstruct RGB and depth views. Second, the architecture design is general enough to cope with two different settings (SS and SS-D) and different modalities. We preferred the ability to work multi-modal and multi-task to an architecture narrowed to processing one particular task. Moreover, since the common representation is complementary to many of the state of art improvements, the proposed architecture can integrate most of them to boost the performance.

Evaluation metrics. To evaluate our network on the segmentation task, we prefer the intersection-over-union (IoU) score to the pixel accuracy. The pixel accuracy is known for being sensitive to the class disbalance, when many images include the large objects such as bed, wall, floor, etc.. The accuracy value may be misleading when the network performs better on the large objects and worse on the small ones. Instead, IoU score remains informative on both balanced and unbalanced datasets.

Let $C_{ij}$ denote the number of pixels those are predicted as class $j$ but actually belongs to class $i$, where $i,j\in L$. Then $C_{ii}$ denotes the number of pixels with correct prediction of class $i$. Let $T_{i}$ denote the total number of pixels that belongs to class $i$ in the ground truth, $K$ is the total number of classes in the dataset. Then IoU is the average value of the intersection between the ground truth and the predictions: $IoU=\frac{1}{K}\sum_{i}\frac{C_{ii}}{T_{i}+\sum_{j}C_{ji}-C_{ii}}.$

For depth estimation, we use the root mean square error (RMSE) that measures the error between the estimated depth and ground truth.

Hyper-parameters. We set $W^{{}^{\prime}}=H^{{}^{\prime}}=65$ for the NYU2 set and $W^{{}^{\prime}}=H^{{}^{\prime}}=46$ for the SUN set. Feature maps generated by modality branches are shaped with $d=256$. The number of hidden variables in the common representation is fixed, $k=128$. During the training with the objective function $\cal L$, weight $\lambda$ of reconstruction loss is 1.

IV-C Semantic segmentation and depth estimation

We consider three different ways to use the architecture presented in Section III to process the RGB-D data.

• •

Independent learning: In this case, each modality branch is trained and tested independently. In the SS setting, we have RGB-SS and D-SS branches; in the SS-D setting, we train and test RGB-SS and D-D branches to provide the baseline performances.

• •

Joint learning: The network is trained in two stages as described in Section III. In SS and SS-D settings, the network is trained to minimize the objective function $\cal L$, with the appropriate depth branch loss $L_{d}$ (see Section 3.4). In either case, we compare them to the baselines obtained with the independent training. In the SS setting, we test the common representation with one or two modalities available at test time, where the semantic segmentation is evaluated using the RGB, depth or both images. In the SS-D setting, we test the semantic segmentation and depth estimation with one or two modalities available at test time.

Table I reports IoU values for the SS and SS-D settings on NYU2 dataset. In the SS setting, training two modality branches independently yields 53.1 (RGB) and 37.1 (depth) IoU values; this reflects the RGB view being more informative than depth. Learning a joint model and using the common representation at test time improves the performance in cases when depth or both views are available. As both views address the segmentation task, the common representation makes performance dependent on which view is available at test time. Instead it does not depend which view is being reconstructed.

In the SS-D setting, the baseline for RGB-SS branch is the same, the baseline for depth reconstruction using D-D branch gives RMSE value of 0.51. The common representation improves the RGB value to 54.3, and reduces the reconstruction error to 0.39 and 0.53 when using the depth only or both views, respectively. In the cross-view reconstruction, using depth for segmentation drops IoU value to 35.0 only, using RGB for depth estimation yields 0.72 error.

Table I also reports evaluation results on SUN dataset. Using both views does improve the performance, moreover depth estimation benefits more from the common representation than the segmentation task.

We compare our results to the state of art on four typical scenarios for RGB-D images (see Table II). Our architecture is the only one able to cope with in all cases. moreover it remains competitive to the highly specialized architectures [20, 15] which cope with one or two scenarios only.

IV-D Discussion

Both quantitative and qualitative results validated the effectiveness of learning the common representation from RGB and depth images. However the conducted experiments left some questions open; we discuss them in this section.

In addition to the results reported in Tables I and II, we tested a number of alternatives and made some conclusions. First, adding the view correlation (see Section III-D) does not seem to improve the common representation nor the performance. Second, the scale-irrelevant loss for the depth estimation, mentioned in Section III-B, does not seem to perform better than the standard $L_{2}$ and smoothed $L_{1}$ losses; all SS-D results in table I and II refer to the smoothed $L_{1}$ loss.

The two-stage training of the network enables to play with a so-called frozen configuration. The modality branches trained at the first stage get frozen and extract feature maps for all RGB-D images in the dataset. Such a frozen configuration allowed to test different configurations of common representation network before training the full network end-to-end. Below we finally mention some ideas on further improving the current architecture.

The common representation is currently limited to one hidden layer. Using deeper multi-view autoencoders has been beneficial in the frozen case. 2. 2.

Learning the common representation is implemented on one fixed scale ($k=128$) of the RGB and depth feature maps. We consider replacing one-fixed-scale MAE with multi-scale ones, on each level of the encoder-decoder networks. 3. 3.

ResNet101 model pre-trained in COCO dataset fits well the segmentation task, but to the less extend the depth estimation task. We consider setting up a more appropriate pre-trained model or an option of training it from scratch or combine the two models [15].

V CONCLUSION

We proposed a new deep learning architecture for the tasks of semantic segmentation and depth prediction from RGB-D images. In the proposed architecture, the conventional feature fusion is replaced with a common deep representation of the RGB and depth views. Combined with an encoder-decoder type of the network, the architecture allows for a joint learning for the semantic segmentation and depth estimation based on their common representation. This approach offers several important advantages, such as using one modality at test time to build a common representation and to reconstruct the missing modality. We reported a number of evaluation results on two standard RGB-D datasets. Both quantitative and qualitative results validated the effectiveness of learning the common representation from RGB and depth images.