HAMMER: Heterogeneous, Multi-Robot Semantic Gaussian Splatting

TL;DR

HAMMER is a collaborative, server-based method that constructs high-fidelity, semantic 3D maps from asynchronous multi-robot data streams without prior pose knowledge, enabling advanced navigation and scene understanding.

Contribution

It introduces a novel framework for real-time, multi-robot 3D semantic mapping using Gaussian Splatting with no initial pose information and supports open-vocabulary language queries.

Findings

Creates maps with twice the fidelity of baselines

Enables semantic goal-conditioned navigation tasks

Handles diverse perception modes and device variations

Abstract

3D Gaussian Splatting offers expressive scene reconstruction, modeling a broad range of visual, geometric, and semantic information. However, efficient real-time map reconstruction with data streamed from multiple robots and devices remains a challenge. To that end, we propose HAMMER, a server-based collaborative Gaussian Splatting method that leverages widely available ROS communication infrastructure to generate 3D, metric-semantic maps from asynchronous robot data-streams with no prior knowledge of initial robot positions and varying on-device pose estimators. HAMMER consists of (i) a frame alignment module that transforms local SLAM poses and image data into a global frame and requires no prior relative pose knowledge, and (ii) an online module for training semantic 3DGS maps from streaming data. HAMMER handles mixed perception modes, adjusts automatically for variations in image pre-processing among different devices, and distills CLIP semantic codes into the 3D scene for open-vocabulary language queries. In our real-world experiments, HAMMER creates higher-fidelity maps (2x) compared to competing baselines and is useful for downstream tasks, such as semantic goal-conditioned navigation (e.g., "go to the couch"). Accompanying content available at hammer-project.github.io.