Synthetic-to-Real Domain Generalized Semantic Segmentation for 3D Indoor Point Clouds

TL;DR

This paper introduces a synthetic-to-real domain generalization framework for 3D indoor point cloud semantic segmentation, using novel augmentation and prototype techniques to improve performance on real-world data.

Contribution

It proposes CINMix augmentation and multi-prototypes to address domain gaps caused by layout and point pattern differences.

Findings

Enhanced generalization to real-world datasets

CINMix diversifies source data layouts

Multi-prototypes model intra-class variance effectively

Abstract

Semantic segmentation in 3D indoor scenes has achieved remarkable performance under the supervision of large-scale annotated data. However, previous works rely on the assumption that the training and testing data are of the same distribution, which may suffer from performance degradation when evaluated on the out-of-distribution scenes. To alleviate the annotation cost and the performance degradation, this paper introduces the synthetic-to-real domain generalization setting to this task. Specifically, the domain gap between synthetic and real-world point cloud data mainly lies in the different layouts and point patterns. To address these problems, we first propose a clustering instance mix (CINMix) augmentation technique to diversify the layouts of the source data. In addition, we augment the point patterns of the source data and introduce non-parametric multi-prototypes to ameliorate the intra-class variance enlarged by the augmented point patterns. The multi-prototypes can model the intra-class variance and rectify the global classifier in both training and inference stages. Experiments on the synthetic-to-real benchmark demonstrate that both CINMix and multi-prototypes can narrow the distribution gap and thus improve the generalization ability on real-world datasets.