Open-Vocabulary High-Resolution 3D (OVHR3D) Data Segmentation and Annotation Framework

TL;DR

This paper introduces OVHR3D, a framework that combines advanced models and visualization tools to improve 3D data segmentation and annotation efficiency for military simulation environments.

Contribution

It presents a novel integrated framework utilizing Grounding DINO, Segment Anything Model, and enhanced 2D rendering for efficient 3D data annotation.

Findings

Enhanced annotation efficiency demonstrated

Framework effectively integrates multiple models

User-friendly interface facilitates annotation process

Abstract

In the domain of the U.S. Army modeling and simulation, the availability of high quality annotated 3D data is pivotal to creating virtual environments for training and simulations. Traditional methodologies for 3D semantic and instance segmentation, such as KpConv, RandLA, Mask3D, etc., are designed to train on extensive labeled datasets to obtain satisfactory performance in practical tasks. This requirement presents a significant challenge, given the inherent scarcity of manually annotated 3D datasets, particularly for the military use cases. Recognizing this gap, our previous research leverages the One World Terrain data repository manually annotated databases, as showcased at IITSEC 2019 and 2021, to enrich the training dataset for deep learning models. However, collecting and annotating large scale 3D data for specific tasks remains costly and inefficient. To this end, the objective of this research is to design and develop a comprehensive and efficient framework for 3D segmentation tasks to assist in 3D data annotation. This framework integrates Grounding DINO and Segment anything Model, augmented by an enhancement in 2D image rendering via 3D mesh. Furthermore, the authors have also developed a user friendly interface that facilitates the 3D annotation process, offering intuitive visualization of rendered images and the 3D point cloud.