Learning 3D Representations for Spatial Intelligence from Unposed Multi-View Images

TL;DR

This paper introduces UniSplat, a novel feed-forward framework for learning robust 3D representations from unposed multi-view images, addressing geometry, appearance, and semantic inconsistencies.

Contribution

UniSplat combines dual masking, coarse-to-fine Gaussian splatting, and pose-conditioned recalibration to improve 3D representation learning from unposed images.

Findings

Enhances geometry induction in unposed multi-view images.

Reduces appearance and semantic inconsistencies through progressive refinement.

Enforces geometric-semantic consistency for robust 3D representations.

Abstract

Robust 3D representation learning forms the perceptual foundation of spatial intelligence, enabling downstream tasks in scene understanding and embodied AI. However, learning such representations directly from unposed multi-view images remains challenging. Recent self-supervised methods attempt to unify geometry, appearance, and semantics in a feed-forward manner, but they often suffer from weak geometry induction, limited appearance detail, and inconsistencies between geometry and semantics. We introduce UniSplat, a feed-forward framework designed to address these limitations through three complementary components. First, we propose a dual-masking strategy that strengthens geometry induction in the encoder. By masking both encoder and decoder tokens, and targeting decoder masks toward geometry-rich regions, the model is forced to infer structural information from incomplete visual cues, yielding geometry-aware representations even under unposed inputs. Second, we develop a coarse-to-fine Gaussian splatting strategy that reduces appearance-semantics inconsistencies by progressively refining the radiance field. Finally, to enforce geometric-semantic consistency, we introduce a pose-conditioned recalibration mechanism that interrelates the outputs of multiple heads by re-projecting predicted 3D point and semantic maps into the image plane using estimated camera parameters, and aligning them with corresponding RGB and semantic predictions to ensure cross-task consistency, thereby resolving geometry-semantic mismatches. Together, these components yield unified 3D representations that are robust to unposed, sparse-view inputs and generalize across diverse tasks, laying a perceptual foundation for spatial intelligence.