TALO: Pushing 3D Vision Foundation Models Towards Globally Consistent Online Reconstruction

TL;DR

This paper introduces TALO, a novel alignment framework for 3D vision foundation models that enhances temporal consistency and robustness in online 3D reconstruction tasks across various models and camera setups.

Contribution

We propose a higher-DOF, long-term alignment method using Thin Plate Spline and a point-agnostic registration approach, improving consistency and robustness in online 3D reconstruction.

Findings

Achieves more coherent geometry and lower trajectory errors

Demonstrates robustness across multiple datasets and models

Compatible with diverse camera configurations

Abstract

3D vision foundation models have shown strong generalization in reconstructing key 3D attributes from uncalibrated images through a single feed-forward pass. However, when deployed in online settings such as driving scenarios, predictions are made over temporal windows, making it non-trivial to maintain consistency across time. Recent strategies align consecutive predictions by solving global transformation, yet our analysis reveals their fundamental limitations in assumption validity, local alignment scope, and robustness under noisy geometry. In this work, we propose a higher-DOF and long-term alignment framework based on Thin Plate Spline, leveraging globally propagated control points to correct spatially varying inconsistencies. In addition, we adopt a point-agnostic submap registration design that is inherently robust to noisy geometry predictions. The proposed framework is fully plug-and-play, compatible with diverse 3D foundation models and camera configurations (e.g., monocular or surround-view). Extensive experiments demonstrate that our method consistently yields more coherent geometry and lower trajectory errors across multiple datasets, backbone models, and camera setups, highlighting its robustness and generality. Code is available at https://github.com/Xian-Bei/TALO.