Voxel-Aggregated Feature Synthesis: Efficient Dense Mapping for Simulated 3D Reasoning

TL;DR

VAFS is a novel method that significantly accelerates dense 3D mapping in simulation by synthesizing features from segmented point clouds, reducing redundant computation and improving accuracy.

Contribution

The paper introduces VAFS, a new approach that reduces computational load in dense 3D mapping by synthesizing features, enabling faster and more accurate semantic mapping in simulation.

Findings

VAFS outperforms prior methods in speed and accuracy.

Semantic IoU scores are higher with VAFS.

VAFS reduces feature embedding from frames to objects.

Abstract

We address the issue of the exploding computational requirements of recent State-of-the-art (SOTA) open set multimodel 3D mapping (dense 3D mapping) algorithms and present Voxel-Aggregated Feature Synthesis (VAFS), a novel approach to dense 3D mapping in simulation. Dense 3D mapping involves segmenting and embedding sequential RGBD frames which are then fused into 3D. This leads to redundant computation as the differences between frames are small but all are individually segmented and embedded. This makes dense 3D mapping impractical for research involving embodied agents in which the environment, and thus the mapping, must be modified with regularity. VAFS drastically reduces this computation by using the segmented point cloud computed by a simulator's physics engine and synthesizing views of each region. This reduces the number of features to embed from the number of captured RGBD frames to the number of objects in the scene, effectively allowing a "ground truth" semantic map to be computed an order of magnitude faster than traditional methods. We test the resulting representation by assessing the IoU scores of semantic queries for different objects in the simulated scene, and find that VAFS exceeds the accuracy and speed of prior dense 3D mapping techniques.