APNet: Urban-level Scene Segmentation of Aerial Images and Point Clouds

TL;DR

APNet introduces a dual-branch network architecture that combines point cloud and aerial image data for urban scene segmentation, achieving state-of-the-art results by effectively fusing diverse scene representations.

Contribution

The paper presents a novel dual-branch network with a geometry-aware fusion module for improved urban scene segmentation from point clouds and aerial images.

Findings

Fusion output outperforms individual branches

Achieves 65.2 mIoU on SensatUrban dataset

State-of-the-art performance in urban scene segmentation

Abstract

In this paper, we focus on semantic segmentation method for point clouds of urban scenes. Our fundamental concept revolves around the collaborative utilization of diverse scene representations to benefit from different context information and network architectures. To this end, the proposed network architecture, called APNet, is split into two branches: a point cloud branch and an aerial image branch which input is generated from a point cloud. To leverage the different properties of each branch, we employ a geometry-aware fusion module that is learned to combine the results of each branch. Additional separate losses for each branch avoid that one branch dominates the results, ensure the best performance for each branch individually and explicitly define the input domain of the fusion network assuring it only performs data fusion. Our experiments demonstrate that the fusion output consistently outperforms the individual network branches and that APNet achieves state-of-the-art performance of 65.2 mIoU on the SensatUrban dataset. Upon acceptance, the source code will be made accessible.