MapBERT: Bitwise Masked Modeling for Real-Time Semantic Mapping Generation

TL;DR

MapBERT introduces a bitwise masked modeling framework using a lookup-free BitVAE and transformer architecture to generate complete semantic maps in real time, improving indoor spatial awareness for embodied agents.

Contribution

It pioneers the use of bitwise encoding with BitVAE and a masked transformer for semantic map completion, incorporating object-aware masking for better reasoning in unseen spaces.

Findings

Achieves state-of-the-art semantic map generation on Gibson benchmarks.

Balances computational efficiency with accurate unobserved region reconstruction.

Effectively models indoor semantic distributions for robotic applications.

Abstract

Spatial awareness is a critical capability for embodied agents, as it enables them to anticipate and reason about unobserved regions. The primary challenge arises from learning the distribution of indoor semantics, complicated by sparse, imbalanced object categories and diverse spatial scales. Existing methods struggle to robustly generate unobserved areas in real time and do not generalize well to new environments. To this end, we propose \textbf{MapBERT}, a novel framework designed to effectively model the distribution of unseen spaces. Motivated by the observation that the one-hot encoding of semantic maps aligns naturally with the binary structure of bit encoding, we, for the first time, leverage a lookup-free BitVAE to encode semantic maps into compact bitwise tokens. Building on this, a masked transformer is employed to infer missing regions and generate complete semantic maps from limited observations. To enhance object-centric reasoning, we propose an object-aware masking strategy that masks entire object categories concurrently and pairs them with learnable embeddings, capturing implicit relationships between object embeddings and spatial tokens. By learning these relationships, the model more effectively captures indoor semantic distributions crucial for practical robotic tasks. Experiments on Gibson benchmarks show that MapBERT achieves state-of-the-art semantic map generation, balancing computational efficiency with accurate reconstruction of unobserved regions.