Learning and Memorizing Representative Prototypes for 3D Point Cloud Semantic and Instance Segmentation

TL;DR

This paper introduces a memory-augmented network that learns and memorizes representative prototypes to improve 3D point cloud semantic and instance segmentation, especially for non-dominant and rare cases, enhancing accuracy and generalization.

Contribution

The paper proposes a novel memory module that alleviates forgetting by storing diverse pattern prototypes, improving segmentation performance on imbalanced 3D point cloud data.

Findings

Significant accuracy improvements on S3DIS and ScanNetV2 benchmarks.

Enhanced performance on non-dominant classes and rare cases.

Efficient retrieval of meaningful prototypes improves generalization.

Abstract

3D point cloud semantic and instance segmentation is crucial and fundamental for 3D scene understanding. Due to the complex structure, point sets are distributed off balance and diversely, which appears as both category imbalance and pattern imbalance. As a result, deep networks can easily forget the non-dominant cases during the learning process, resulting in unsatisfactory performance. Although re-weighting can reduce the influence of the well-classified examples, they cannot handle the non-dominant patterns during the dynamic training. In this paper, we propose a memory-augmented network to learn and memorize the representative prototypes that cover diverse samples universally. Specifically, a memory module is introduced to alleviate the forgetting issue by recording the patterns seen in mini-batch training. The learned memory items consistently reflect the interpretable and meaningful information for both dominant and non-dominant categories and cases. The distorted observations and rare cases can thus be augmented by retrieving the stored prototypes, leading to better performances and generalization. Exhaustive experiments on the benchmarks, i.e. S3DIS and ScanNetV2, reflect the superiority of our method on both effectiveness and efficiency. Not only the overall accuracy but also nondominant classes have improved substantially.