Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs

TL;DR

This paper introduces an equation-anchored framework for 3D localization in multimodal large language models that explicitly incorporates camera parameters, improving robustness across varying camera intrinsics.

Contribution

It proposes a novel tool-use approach that embeds camera equations into the model's reasoning process, enabling deterministic propagation of camera information.

Findings

Outperforms RGB-only and tool-augmented baselines across varying camera scales.

Significant improvements when camera intrinsics deviate from training conditions.

Explicitly writing the pinhole projection equation enhances 3D localization accuracy.

Abstract

3D localization in Multimodal Large Language Models (MLLMs), including 3D object detection and 3D visual grounding, is fundamentally limited by camera intrinsic ambiguity: the same image admits different 3D scenes under different cameras. Existing MLLMs either ignore camera parameters and overfit to a canonical training intrinsic, or retrieve depth and 3D cues from external tools but treat the returned values as reference cues (numerical hints that the model is free to interpret implicitly), both preventing camera information from being deterministically propagated into the prediction. We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables. The proposed framework proactively retrieves camera intrinsics and samples multi-point metric depths, writes the pinhole back-projection equation $\hat{X} = (u_c - c_x)\bar{Z}/f_x$ explicitly in Chain-of-Thought (CoT), and substitutes tool outputs into the formula before regressing the final 9-DoF bounding box. On both 3D object detection and 3D visual grounding tasks under rescaled camera intrinsics from $0.5\times$ to $1.5\times$, our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale. Code and data will be released.