Lightweight integration of 3D features to improve 2D image segmentation

TL;DR

This paper introduces a lightweight method to incorporate 3D geometric features into 2D image segmentation networks, enhancing performance without needing 3D groundtruth data or significant additional training resources.

Contribution

The proposed approach enables 2D segmentation models to leverage 3D information through joint training with a lightweight 3D feature extractor, avoiding the need for 3D labels.

Findings

Significant performance improvements in 2D segmentation accuracy.

No requirement for 3D groundtruth labels during training.

Lightweight integration with minimal increase in model complexity.

Abstract

Scene understanding has made tremendous progress over the past few years, as data acquisition systems are now providing an increasing amount of data of various modalities (point cloud, depth, RGB...). However, this improvement comes at a large cost on computation resources and data annotation requirements. To analyze geometric information and images jointly, many approaches rely on both a 2D loss and 3D loss, requiring not only 2D per pixel-labels but also 3D per-point labels. However, obtaining a 3D groundtruth is challenging, time-consuming and error-prone. In this paper, we show that image segmentation can benefit from 3D geometric information without requiring a 3D groundtruth, by training the geometric feature extraction and the 2D segmentation network jointly, in an end-to-end fashion, using only the 2D segmentation loss. Our method starts by extracting a map of 3D features directly from a provided point cloud by using a lightweight 3D neural network. The 3D feature map, merged with the RGB image, is then used as an input to a classical image segmentation network. Our method can be applied to many 2D segmentation networks, improving significantly their performance with only a marginal network weight increase and light input dataset requirements, since no 3D groundtruth is required.