Multi-task 3D building understanding with multi-modal pretraining

TL;DR

This paper presents a multi-task learning approach using multi-modal pretraining for 3D building classification and segmentation, achieving state-of-the-art results and winning the CVPR23 BuildingNet challenge.

Contribution

It extends ULIP with PointNeXt for multi-task learning on BuildingNet, demonstrating significant improvements in accuracy and segmentation metrics.

Findings

Achieved 59.36% accuracy in 3D building type classification.

Obtained 31.68 PartIoU in building part segmentation.

Won 1st place in the CVPR23 BuildingNet challenge.

Abstract

This paper explores various learning strategies for 3D building type classification and part segmentation on the BuildingNet dataset. ULIP with PointNeXt and PointNeXt segmentation are extended for the classification and segmentation task on BuildingNet dataset. The best multi-task PointNeXt-s model with multi-modal pretraining achieves 59.36 overall accuracy for 3D building type classification, and 31.68 PartIoU for 3D building part segmentation on validation split. The final PointNeXt XL model achieves 31.33 PartIoU and 22.78 ShapeIoU on test split for BuildingNet-Points segmentation, which significantly improved over PointNet++ model reported from BuildingNet paper, and it won the 1st place in the BuildingNet challenge at CVPR23 StruCo3D workshop.