Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiation

TL;DR

Allocentric Perceiver is a training-free method that improves spatial reasoning in vision-language models by explicitly reconstructing 3D geometry and aligning reference frames, leading to significant gains on allocentric tasks.

Contribution

It introduces a novel, training-free approach that disentangles allocentric reasoning from egocentric priors by using geometric experts and frame instantiation.

Findings

Achieves ~10% improvement on allocentric spatial reasoning benchmarks.

Maintains strong egocentric performance.

Outperforms existing spatial perception models.

Abstract

With the rising need for spatially grounded tasks such as Vision-Language Navigation/Action, allocentric perception capabilities in Vision-Language Models (VLMs) are receiving growing focus. However, VLMs remain brittle on allocentric spatial queries that require explicit perspective shifts, where the answer depends on reasoning in a target-centric frame rather than the observed camera view. Thus, we introduce Allocentric Perceiver, a training-free strategy that recovers metric 3D states from one or more images with off-the-shelf geometric experts, and then instantiates a query-conditioned allocentric reference frame aligned with the instruction's semantic intent. By deterministically transforming reconstructed geometry into the target frame and prompting the backbone VLM with structured, geometry-grounded representations, Allocentric Perceriver offloads mental rotation from implicit reasoning to explicit computation. We evaluate Allocentric Perciver across multiple backbone families on spatial reasoning benchmarks, observing consistent and substantial gains ($\sim$10%) on allocentric tasks while maintaining strong egocentric performance, and surpassing both spatial-perception-finetuned models and state-of-the-art open-source and proprietary models.