KitchenTwin: Semantically and Geometrically Grounded 3D Kitchen Digital Twins

TL;DR

This paper introduces a scale-aware 3D fusion framework for creating accurate, metrically consistent digital twins of indoor kitchens by combining transformer-based point clouds with object meshes, guided by vision-language models.

Contribution

It presents a novel geometric anchor mechanism and registration pipeline that resolve scale and coordinate mismatches for improved digital twin accuracy.

Findings

Enhanced object alignment and geometric consistency in real indoor environments

Improved performance in multi-primitive fitting and metric measurement tasks

Open-source dataset with metrically scaled, semantically grounded 3D scenes

Abstract

Embodied AI training and evaluation require object-centric digital twin environments with accurate metric geometry and semantic grounding. Recent transformer-based feedforward reconstruction methods can efficiently predict global point clouds from sparse monocular videos, yet these geometries suffer from inherent scale ambiguity and inconsistent coordinate conventions. This mismatch prevents the reliable fusion of these dimensionless point cloud predictions with locally reconstructed object meshes. We propose a novel scale-aware 3D fusion framework that registers visually grounded object meshes with transformer-predicted global point clouds to construct metrically consistent digital twins. Our method introduces a Vision-Language Model (VLM)-guided geometric anchor mechanism that resolves this fundamental coordinate mismatch by recovering an accurate real-world metric scale. To fuse these networks, we propose a geometry-aware registration pipeline that explicitly enforces physical plausibility through gravity-aligned vertical estimation, Manhattan-world structural constraints, and collision-free local refinement. Experiments on real indoor kitchen environments demonstrate improved cross-network object alignment and geometric consistency for downstream tasks, including multi-primitive fitting and metric measurement. We additionally introduce an open-source indoor digital twin dataset with metrically scaled scenes and semantically grounded and registered object-centric mesh annotations.