Sparsity Invariant CNNs

TL;DR

This paper introduces a sparse convolutional neural network architecture that effectively processes sparse data, particularly for depth upsampling from laser scans, outperforming traditional CNNs on sparse inputs and generalizing well across datasets.

Contribution

The paper proposes a novel sparse convolution layer that explicitly handles missing data locations, improving CNN performance on sparse inputs and providing a new dataset for depth estimation tasks.

Findings

Sparse CNNs outperform traditional CNNs on sparse data

The proposed network generalizes across datasets and sparsity levels

A new KITTI-based dataset for depth upsampling is introduced

Abstract

In this paper, we consider convolutional neural networks operating on sparse inputs with an application to depth upsampling from sparse laser scan data. First, we show that traditional convolutional networks perform poorly when applied to sparse data even when the location of missing data is provided to the network. To overcome this problem, we propose a simple yet effective sparse convolution layer which explicitly considers the location of missing data during the convolution operation. We demonstrate the benefits of the proposed network architecture in synthetic and real experiments with respect to various baseline approaches. Compared to dense baselines, the proposed sparse convolution network generalizes well to novel datasets and is invariant to the level of sparsity in the data. For our evaluation, we derive a novel dataset from the KITTI benchmark, comprising 93k depth annotated RGB images. Our dataset allows for training and evaluating depth upsampling and depth prediction techniques in challenging real-world settings and will be made available upon publication.