Point-MoE: Large-Scale Multi-Dataset Training with Mixture-of-Experts for 3D Semantic Segmentation

TL;DR

This paper introduces Point-MoE, a scalable Mixture-of-Experts model for 3D semantic segmentation trained on multiple datasets without dataset labels, improving performance and generalization across diverse 3D point cloud data.

Contribution

The paper proposes Point-MoE, a novel sparse Mixture-of-Experts architecture that enables large-scale multi-dataset training for 3D segmentation without dataset supervision.

Findings

Outperforms prior methods on seen datasets

Effective in zero-shot generalization to new datasets

Demonstrates scalable training on heterogeneous 3D data

Abstract

While massively scaling both data and models have become central in NLP and 2D vision, their benefits for 3D point cloud understanding remain limited. We study the initial step of scaling 3D point cloud understanding under a realistic regime: large-scale multi-dataset joint training for 3D semantic segmentation, with no dataset labels available at training or inference time. Point clouds arise from a wide range of sensors (e.g., depth cameras, LiDAR) and scenes (\eg, indoor, outdoor), yielding heterogeneous scanning patterns, sampling densities, and semantic biases; naively mixing such datasets degrades standard models. Therefore, we introduce Point-MoE, a Mixture-of-Experts design that expands model capacity through sparsely activated expert MLPs and a lightweight top-$k$ router, allowing tokens to select specialized experts without requiring dataset supervision. Trained jointly on a diverse mix of indoor and outdoor datasets, and evaluated on seen datasets as well as in zero-shot settings, Point-MoE outperforms prior methods without using dataset labels for either training or inference. This outlines a scalable path for 3D perception: letting the model discover structure in heterogeneous 3D data rather than imposing it via manual curation or dataset-specific heuristics.