SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs

TL;DR

This paper introduces SoPE, a spherical coordinate-based positional embedding for 3D LVLMs, improving spatial perception by better capturing 3D structures and directional information, leading to enhanced multimodal understanding.

Contribution

We propose SoPE, a novel spherical coordinate-based positional embedding that better preserves 3D spatial and angular information for 3D LVLMs, surpassing traditional RoPE methods.

Findings

Improved spatial awareness in 3D LVLMs.

Enhanced performance on multiple 3D scene benchmarks.

Strong generalization demonstrated in real-world tests.

Abstract

3D Large Vision-Language Models (3D LVLMs) built upon Large Language Models (LLMs) have achieved remarkable progress across various multimodal tasks. However, their inherited position-dependent modeling mechanism, Rotary Position Embedding (RoPE), remains suboptimal for 3D multimodal understanding. The vanilla RoPE formulation fails to preserve essential three-dimensional spatial structures when encoding 3D tokens, and its relative distance computation overlooks angular dependencies, hindering the model's ability to capture directional variations in visual representations. To overcome these limitations, we introduce Spherical Coordinate-based Positional Embedding (SoPE). Our method maps point-cloud token indices into a 3D spherical coordinate space, enabling unified modeling of spatial locations and directional angles. This formulation preserves the inherent geometric structure of point-cloud data, enhances spatial awareness, and yields more consistent and expressive geometric representations for multimodal learning. In addition, we introduce a multi-scale frequency mixing strategy to fuse feature information across different frequency domains. Experimental results on multiple 3D scene benchmarks validate the effectiveness of our approach, while real-world deployment experiments further demonstrate its strong generalization capability.