Point Cloud Segmentation of Agricultural Vehicles using 3D Gaussian Splatting

TL;DR

This paper presents a synthetic data generation pipeline using 3D Gaussian Splatting for agricultural vehicle segmentation, achieving high accuracy with models trained solely on synthetic data, reducing reliance on costly real datasets.

Contribution

Introduces a novel synthetic data generation pipeline leveraging 3D Gaussian Splatting and Gaussian Opacity Fields for agricultural vehicle point cloud segmentation.

Findings

Point Transformer V3 achieved 91.35% mIoU trained only on synthetic data.

Models trained on synthetic data sometimes outperformed those trained on real data.

Models demonstrated cross-class generalization to unseen mesh models.

Abstract

Training neural networks for tasks such as 3D point cloud semantic segmentation demands extensive datasets, yet obtaining and annotating real-world point clouds is costly and labor-intensive. This work aims to introduce a novel pipeline for generating realistic synthetic data, by leveraging 3D Gaussian Splatting (3DGS) and Gaussian Opacity Fields (GOF) to generate 3D assets of multiple different agricultural vehicles instead of using generic models. These assets are placed in a simulated environment, where the point clouds are generated using a simulated LiDAR. This is a flexible approach that allows changing the LiDAR specifications without incurring additional costs. We evaluated the impact of synthetic data on segmentation models such as PointNet++, Point Transformer V3, and OACNN, by training and validating the models only on synthetic data. Remarkably, the PTv3 model had an mIoU of 91.35\%, a noteworthy result given that the model had neither been trained nor validated on any real data. Further studies even suggested that in certain scenarios the models trained only on synthetically generated data performed better than models trained on real-world data. Finally, experiments demonstrated that the models can generalize across semantic classes, enabling accurate predictions on mesh models they were never trained on.