Retro-FPN: Retrospective Feature Pyramid Network for Point Cloud Semantic Segmentation

TL;DR

Retro-FPN introduces a retrospective feature refinement process using a retro-transformer to enhance per-point semantic features in point cloud segmentation, significantly improving accuracy over existing methods.

Contribution

The paper proposes Retro-FPN, a novel retro-transformer based module that explicitly refines per-point features by modeling the entire feature pyramid, improving semantic segmentation performance.

Findings

Retro-FPN outperforms state-of-the-art methods on benchmark datasets.

The retro-transformer effectively summarizes semantic context from previous layers.

Retro-FPN is compatible with various backbone architectures.

Abstract

Learning per-point semantic features from the hierarchical feature pyramid is essential for point cloud semantic segmentation. However, most previous methods suffered from ambiguous region features or failed to refine per-point features effectively, which leads to information loss and ambiguous semantic identification. To resolve this, we propose Retro-FPN to model the per-point feature prediction as an explicit and retrospective refining process, which goes through all the pyramid layers to extract semantic features explicitly for each point. Its key novelty is a retro-transformer for summarizing semantic contexts from the previous layer and accordingly refining the features in the current stage. In this way, the categorization of each point is conditioned on its local semantic pattern. Specifically, the retro-transformer consists of a local cross-attention block and a semantic gate unit. The cross-attention serves to summarize the semantic pattern retrospectively from the previous layer. And the gate unit carefully incorporates the summarized contexts and refines the current semantic features. Retro-FPN is a pluggable neural network that applies to hierarchical decoders. By integrating Retro-FPN with three representative backbones, including both point-based and voxel-based methods, we show that Retro-FPN can significantly improve performance over state-of-the-art backbones. Comprehensive experiments on widely used benchmarks can justify the effectiveness of our design. The source is available at https://github.com/AllenXiangX/Retro-FPN