Text-guided Synthetic Geometric Augmentation for Zero-shot 3D Understanding

TL;DR

This paper introduces TeGA, a text-guided geometric augmentation method that generates synthetic 3D data to improve zero-shot 3D classification, achieving state-of-the-art results with limited real data.

Contribution

TeGA is a novel approach that uses generative text-to-3D models and a filtering strategy to expand limited 3D datasets for zero-shot recognition.

Findings

Achieves 3.0% improvement on Objaverse-LVIS

Achieves 4.6% improvement on ScanObjectNN

Achieves 8.7% improvement on ModelNet40

Abstract

Zero-shot recognition models require extensive training data for generalization. However, in zero-shot 3D classification, collecting 3D data and captions is costly and laborintensive, posing a significant barrier compared to 2D vision. Recent advances in generative models have achieved unprecedented realism in synthetic data production, and recent research shows the potential for using generated data as training data. Here, naturally raising the question: Can synthetic 3D data generated by generative models be used as expanding limited 3D datasets? In response, we present a synthetic 3D dataset expansion method, Textguided Geometric Augmentation (TeGA). TeGA is tailored for language-image-3D pretraining, which achieves SoTA in zero-shot 3D classification, and uses a generative textto-3D model to enhance and extend limited 3D datasets. Specifically, we automatically generate text-guided synthetic 3D data and introduce a consistency filtering strategy to discard noisy samples where semantics and geometric shapes do not match with text. In the experiment to double the original dataset size using TeGA, our approach demonstrates improvements over the baselines, achieving zeroshot performance gains of 3.0% on Objaverse-LVIS, 4.6% on ScanObjectNN, and 8.7% on ModelNet40. These results demonstrate that TeGA effectively bridges the 3D data gap, enabling robust zero-shot 3D classification even with limited real training data and paving the way for zero-shot 3D vision application.