SeqVLM: Proposal-Guided Multi-View Sequences Reasoning via VLM for Zero-Shot 3D Visual Grounding

TL;DR

SeqVLM introduces a zero-shot 3D visual grounding framework that uses multi-view images and spatial reasoning to improve object localization without scene-specific training.

Contribution

It proposes a novel multi-view projection and dynamic scheduling approach to enhance zero-shot 3D visual grounding performance.

Findings

Achieves state-of-the-art zero-shot accuracy on ScanRefer and Nr3D benchmarks.

Surpasses previous zero-shot methods by over 4% in accuracy.

Demonstrates improved generalization for real-world 3D visual grounding.

Abstract

3D Visual Grounding (3DVG) aims to localize objects in 3D scenes using natural language descriptions. Although supervised methods achieve higher accuracy in constrained settings, zero-shot 3DVG holds greater promise for real-world applications since eliminating scene-specific training requirements. However, existing zero-shot methods face challenges of spatial-limited reasoning due to reliance on single-view localization, and contextual omissions or detail degradation. To address these issues, we propose SeqVLM, a novel zero-shot 3DVG framework that leverages multi-view real-world scene images with spatial information for target object reasoning. Specifically, SeqVLM first generates 3D instance proposals via a 3D semantic segmentation network and refines them through semantic filtering, retaining only semantic-relevant candidates. A proposal-guided multi-view projection strategy then projects these candidate proposals onto real scene image sequences, preserving spatial relationships and contextual details in the conversion process of 3D point cloud to images. Furthermore, to mitigate VLM computational overload, we implement a dynamic scheduling mechanism that iteratively processes sequances-query prompts, leveraging VLM's cross-modal reasoning capabilities to identify textually specified objects. Experiments on the ScanRefer and Nr3D benchmarks demonstrate state-of-the-art performance, achieving Acc@0.25 scores of 55.6% and 53.2%, surpassing previous zero-shot methods by 4.0% and 5.2%, respectively, which advance 3DVG toward greater generalization and real-world applicability. The code is available at https://github.com/JiawLin/SeqVLM.

Equations

$$\Phi (\mathbf{P})=\{M_i\mid \sigma (M_i)\ge \theta {\}}_{i=1}^m,$$

$$\mathcal{O}\mathcal{P}\mathcal{T}=\{({\text{ID}}_i,C_i,BBo{x}_i,M_i){\}}_{i=1}^m,$$

$$E_t=f_{text}(T_{target}),$$

$$E_c^{(i)}=f_{text}(C_i),$$

$${\zeta }_i=\frac{E_t\cdot E_c^{(i)}}{\Vert E_t\Vert \cdot \Vert E_c^{(i)}\Vert }.$$

$$C^{\ast }=\underset{C_i\in \mathcal{O}\mathcal{P}\mathcal{T}}{\mathrm{argmax}}{\zeta }_i,$$

$$\mathcal{O}=\{M_i\mid C_i=C^{\ast }{\}}_{i=1}^n.$$

$$P_c$$

$$u=\frac{x_c\cdot f_x}{z_c}+c_x,v=\frac{y_c\cdot f_y}{z_c}+c_y.$$

$$\left|\frac{D(u,v)-z_c}{D(u,v)}\right|\le \tau ,$$

$$\mathcal{R}=[u_{\min },v_{\min },u_{\max },v_{\max }],(u,v)\in p_i^j,$$

Full text

3D Visual Grounding, Multi-View Sequences, Visual-Language Model, Zero-shot Scene Understanding

††copyright: acmlicensed††journalyear: 2025††copyright: acmlicensed††conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland††booktitle: Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland††doi: 10.1145/3746027.3754985††isbn: 979-8-4007-2035-2/2025/10††ccs: Computing methodologies Artificial intelligence††ccs: Computing methodologies Computer vision tasks††ccs: Computing methodologies Scene understanding**footnotetext: Jiawen Lin and Shiran Bian contributed equally to this work.††footnotetext: Yanyun Qu and Yachao Zhang are corresponding authors.

1. Introduction

3D Visual Grounding (3DVG) is a critical task that aims to establish cross-modal alignment between natural language descriptions and target objects in 3D scenes. Given a 3D scene represented as point clouds and a semantically rich textual query(e.g., ”the red chair near the window”), the task requires a model to precisely infer the 3D bounding box coordinates of the referred object. This technology holds significant application value in intelligent human-robot interaction [li2024seeground, liu2024aligning], autonomous driving environmental perception [zhou2022context, 3dvg-trans], and AR/VR systems[zhang2024towards, unal2024four].

Current approaches predominantly rely on fully supervised paradigms [scanrefer, tgnn, sat, mcln, eda, 3dsps, mvt, 3dvlp], achieving localization through training on fixed text-scene paired datasets. Although these methods attain high accuracy in closed datasets, they suffer from two limitations: (1) High annotation costs for 3D bounding boxes restrict training data scale, hindering generalization to real-world scene complexity [huang2024training, hwang2023joint, lin2024exploring]; (2) Strict dependence on annotated data impedes adaptation to open-vocabulary scenarios [smith2023flowcam, deitke2023objaverse].

Zero-shot and open-vocabulary learning provide promising directions to address these limitations [llm-grounder, vlmgrounder, zs3dvg, li2024seeground, openscene]. Zero-shot learning enables models to localize unseen objects without training, while open-vocabulary learning focuses on constructing models capable of processing arbitrary vocabulary. The integration of these paradigms enhances model generalization and broadens application scenarios. Concurrently, capitalizing on the profound semantic and scene-level understanding of large language models (LLMs) [zhang2024earthgpt, wu2024visionllm, chen2023minigpt, wang2023chat, huang2025reason3d]and visual-language models (VLMs)[zheng2023preventing, javed2024cplip, phan2024zeetad, chen2024reasoning3d], researchers are developing zero-shot 3D visual grounding frameworks to tackle open-vocabulary challenges.

Existing zero-shot 3DVG methods primarily fall into two categories: LLM-based and VLM-based methods. LLM-based methods [llm-grounder, zs3dvg] convert textual descriptions into structured programmatic instructions via LLMs, then execute code-based object retrieval in point cloud space. While these methods leveraging LLMs’ superior semantic comprehension to parse complex language structures, they struggle with sparse point clouds’ inherent limitations in capturing color/texture cues and reasoning about intricate spatial relationships. Conversely, as shown in Figure 1(a), previous VLM-based methods [vlmgrounder, li2024seeground] enhance 3D visual grounding by integrating 2D visual features to identify objects in images that best match text descriptions. However, their reliance on single-view renderings may introduce cascading limitations. The absence of 3D geometric constraints leads to spatial biases that misalign 2D projections with 3D coordinates, while single-view perspectives inherently fail to resolve occlusions or capture multi-object contextual relationships. These issues are further compounded by gaps between rendered and real-world imagery, such as chromatic aberrations, texture simplification, and occlusion [yan2024multi, cao2024tparn], which systematically degrade the VLMs’ ability to reason about fine-grained visual-textual correspondences [li2023blip, liu2024unimel, zhang2024prototype]. Collectively, these limitations highlight the critical need for geometrically consistent, multi-view fusion strategies in real-world 3D grounding scenarios.

To address these challenges, we propose SeqVLM, a novel framework that enhances cross-modal alignment for zero-shot 3DVG by leveraging multi-view real-world image sequences. SeqVLM integrates 3D point clouds, multi-view images, and natural language descriptions, utilizing VLM for cross-modal alignment to achieve precise 3D object localization. The overall pipeline of SeqVLM is illustrated in Figure 1(b).

Specifically, our framework uses a 3D semantic segmentation network to extract object proposals from 3D point cloud scenes. To improve accuracy and efficiency, irrelevant proposals are filtered through semantic alignment with the target object category described in the input, retaining only those that match semantically with the specified category. The refined proposals are then projected onto real-world image sequences to generate context-aware regions with multi-view consistency, preserving spatial relationships and details. To address the constraints imposed by VLM input length, we introduce an iterative reasoning mechanism that dynamically schedules inference, optimizing both inference efficiency and localization accuracy. Compared to existing approaches, SeqVLM has two main advantages: First, proposal-guided projection strategy effectively maintains 3D geometric attributes and environmental context, significantly reducing localization errors. Second, by leveraging multi-view real images, SeqVLM enhances spatial understanding and contextual awareness, thereby enhancing the cross-modal alignment of VLMs. These innovations enable robust zero-shot 3DVG in complex environments.

SeqVLM achieves absolute improvements of 4.0% and 5.2% in Acc@0.25 on ScanRefer [scanrefer] and Nr3D [referit3d], respectively, outperforming existing zero-shot methods. Notably, its performance rivals fully supervised approaches on certain metrics.

Overall, our contributions are summarized as follows:

• $\bullet$

We propose SeqVLM, a novel framework that integrates 3D geometric features with 2D visual cues via proposal-guided multi-view projection, enabling accurate cross-modal localization in zero-shot settings.

• $\bullet$

We develop a proposal-guided multi-view projection strategy, which aligns objects semantically relevant to textual descriptions with real-scene images across multiple viewpoints. This ensures precise projection positioning while preserving contextual and detailed scene information.

• $\bullet$

We design an iterative reasoning mechanism to address VLM failure in multi-candidate scenarios through dynamic computational scheduling.

• $\bullet$

We conduct extensive experiments on the ScanRefer and Nr3D datasets, demonstrating the effectiveness of our framework and setting new state-of-the-art performance in zero-shot 3D visual grounding.

2. Related Work

2.1. Supervised 3DVG

Current research in 3D visual grounding predominantly adopts supervised learning paradigms, achieving cross-modal alignment through training on annotated datasets. Existing methodologies can be categorized into two technical frameworks: two-stage and single-stage approaches.

Two-stage methods [scanrefer, referit3d, 3dvg-trans, jain2022bottom, chen2022d] employ a ”detect-then-match” cascade architecture. The first stage generates candidate object proposals via 3D object detection or instance segmentation[liu2021group, jiang2020pointgroup, vu2022softgroup], while the second stage constructs cross-modal alignment modules to refine language-query matching with candidate region features and optimize bounding boxes[3djcg, 3dsps]. In contrast, single-stage methods [eda, wang2024g, mcln] utilize end-to-end architectures to jointly model 3D point clouds and textual features, achieving dense regression predictions of target locations. Notably, SAT [sat] pioneers the integration of 2D image semantics as auxiliary supervision during training, enhancing joint representation learning between 3D point clouds and language via cross-modal alignment, thereby significantly improving localization accuracy. Despite remarkable progress, supervised approaches face two critical bottlenecks: (1) High annotation costs for 3D data restrict dataset scalability, limiting coverage of complex real-world scenarios; (2) Model generalization is constrained by closed-vocabulary settings, hindering adaptation to open-world localization of unseen objects.

2.2. Zero-Shot 3DVG

Zero-shot methods aim to overcome the annotation dependency of supervised paradigms by leveraging the cross-modal reasoning capabilities of pretrained large models, primarily divided into Large Language Model (LLM)-based and Visual-Language Model (VLM)-based methods.

LLM-based methods [llm-grounder, zs3dvg] parse natural language queries through linguistic models, converting them into structured semantic elements or executable program instructions. These methods combine geometric reasoning with commonsense knowledge for zero-shot localization. While effective in handling complex semantic inference, they exhibit limitations in exploiting 3D scene contextual information and effectively integrating fine-grained visual features.

VLM-based methods [vlmgrounder, li2024seeground], leverage the multimodal alignment capabilities of visual-language models, enabling the integration of visual and textual information for enhanced scene understanding. VLM-Grounder [vlmgrounder] proposes dynamically stitching multi-view 2D image sequences, utilizing VLM’s open-vocabulary reasoning to infer 2D object masks, followed by multi-view projection and morphological processing for 3D bounding box reconstruction. SeeGround [li2024seeground] further introduces dynamically rendered 2D images with spatially enriched 3D text descriptions, utilizing a Perspective Adaptation Module for query-aligned views and a Fusion Alignment Module to improve object localization. Despite improved scene context utilization, these methods remain constrained by some key factors: (1) Geometric misalignment errors arising from improper projection methodologies; (2) Loss of spatial information related to viewpoint-specific cues due to single-view constraints; (3) Visual feature distortion due to domain gaps between rendered images and real-world scenes.

3. Methodology

3.1. Task Definition

We consider a 3D scene represented by a colored point cloud $\mathbf{P}\in\mathbb{R}^{N\times 6}$, where each point is characterized by XYZ coordinates and RGB color attributes, along with an image list captured from various viewpoints in real-world scenes, and a textual description $T$ specifying the target object $O^{*}$. Our task focuses on zero-shot 3DVG that localizes the $O^{*}$ in 3D space without requiring scene-specific training or fine-tuning. The image list can be obtained through various sensors including RGB-D sensors with structured light scanners [shen2013layer, micko2025motion], dense SLAM systems [hu2024cg, huang2024photo, li2024rgb] or Time-of-Flight (ToF) sensors [xu2024m, qiao2024rgb].

3.2. Overview

The framework of SeqVLM is illustrated in Figure 2. Given a point cloud scene $\mathbf{P}$, a 3D semantic segmentation network is first employed to perform instance segmentation, followed by confidence-based filtering to extract object categories and corresponding mask features. Then, LLM parses the textual description $T$ to identify the target object category, denoted as $C^{*}$. This category, along with the filtered object categories, is embedded using a text encoder. Based on the cosine similarity matching, a proposal list $\mathcal{O}$ is generated, containing $n$ proposals that are semantically related to the target category. For each proposal $O_{i}\in\mathcal{O}$, all scene images from $\mathbf{P}$ are sampled to form an image list $\mathcal{I}_{i}$. Multi-view projection is applied to annotate the region corresponding to $O_{i}$ in each image within $\mathcal{I}_{i}$. The top $n_{\mathit{frame}}$ images with the largest projected areas are then stitched, generating a vertical image sequence ${S}_{i}$ as the 2D visual representation of $O_{i}$. Finally, using an iterative reasoning mechanism, the proposal sequences $\mathcal{S}=[{S}_{1},{S}_{2},\dots,{S}_{k}]$, together with the textual description $T$, are fed into the VLM for cross-modal reasoning to identify the target object $O^{*}$. By deeply integrating multimodal information, SeqVLM significantly enhances localization accuracy and robustness, offering an effective solution for zero-shot object localization in complex 3D scenes.

3.3. Proposal Selection Module

We propose to employ a text-driven semantic filtering mechanism to refine proposals whose categories are consistent with the target object. Specifically, the module processes the point cloud scene $\mathbf{P}$ using a 3D semantic segmentation network $\Phi$, and retain only those instances with confidence scores exceeding a predefined threshold $\theta$:

[TABLE]

where $M_{i}$ and $\sigma(\cdot)$ denote the instance mask features and confidence function, respectively. The adoption of threshold filtering effectively eliminates the unreliable segmentation result.

Following the dynamic vocabulary mapping in ZSVG3D [zs3dvg], a structured Object Profile Table($\mathcal{OPT}$) can be constructed as:

[TABLE]

where $C_{i}$ represents the object category and $BBox_{i}$ corresponds to the 3D bounding box. Then the text encoder is applied to compute the embedding of proposal category $C_{i}$ and the $T_{\mathit{target}}$ generated by LLM:

[TABLE]

where $f_{\mathit{text}}(\cdot)$ is the text encoder, and the cosine similarity is computed as:

[TABLE]

The target object category is determined by the maximum similarity, which then defines the proposal set $\mathcal{O}$ as:

[TABLE]

[TABLE]

This optimization retains only proposals that share the same category as the target, significantly reducing computational complexity in VLM reasoning.

3.4. Proposal-Guided Multi-View Projection

Since VLM cannot process 3D scenes properly, we propose a multi-view projection method that transforms 3D point clouds into 2D images to adapt to VLM. However, in previous works, VLM-Grounder [vlmgrounder] directly maps 2D detection results to 3D space without considering geometric constraints, and SeeGround [li2024seeground] relies on restricted single-view rendering, resulting in inadequate spatial understanding and detail loss. Both approaches lack the full utilization of proposal-guided images. To address these limitations, our multi-view projection method is guided by proposals to preserve the context of objects.

Proposal-Guided Projection.

For one point belonging to a proposal, its world coordinate can be defined as $P_{w}=[x_{w},y_{w},z_{w}]$, which can be projected into the camera coordinate system of the corresponding view via the homogeneous transformation matrix $T_{\mathit{wc}}\in\mathbb{R}^{4\times 4}$:

[TABLE]

The pixel coordinates $(u,v)$ in the view image are then computed using the intrinsic matrix $\mathbf{K}=\begin{bmatrix}f_{x}&0&c_{x}\\ 0&f_{y}&c_{y}\\ 0&0&1\end{bmatrix}$:

[TABLE]

To ensure projection validity, we introduce a depth consistency verification:

[TABLE]

where $D(u,v)$ denotes the depth measurement at $(u,v)$ in the view image, and $\tau$ serves as the visibility threshold. If $P_{w}$ satisfies the consistency after transformation, it is considered valid and visible in the corresponding view.

Multi-View Sequence Generation.

For each proposal $M_{i}\in\mathcal{O}$, which contains a set of point cloud coordinates, we optimize viewpoint selection by quantifying the projected area. The top $n_{\mathit{frame}}$ images with the largest projected areas are selected from the sampled image set $\mathcal{I}_{i}$, forming the pixel projection set $\mathcal{P}_{i}=\{p_{i}^{1},p_{i}^{2},\dots,p_{i}^{n_{\mathit{frame}}}\}$, where each element represents the pixel projection set of $M_{i}$ from a certain viewpoint.

During the image stitching phase, each pixel set $p_{i}^{j}\in\mathcal{P}_{i}$ is analyzed to extract its minimum bounding rectangle in the view image:

[TABLE]

which represents the 2D bounding box of the proposal. To preserve the image details of the object within the bounding box, the box is expanded by the same proportion $\alpha$ along both the width and height directions, and is then highlighted by a distinct 3-pixel-wide red rectangular annotation.

After that, $n_{\mathit{frame}}$ annotated images are vertically concatenated to generate an enhanced image sequence $S_{i}$. This process avoids the location deviation during grounding, and enriches spatial relationships and contextual details of object through multiple viewpoints, which are essential for robust 3D scene understanding by VLM.

3.5. VLM Iterative Reasoning Module

We propose an iterative reasoning mechanism to address the sensitivity of VLM to input length in zero-shot 3DVG tasks. Existing works [vlmgrounder, zhao2024stitch, wang2024survey] indicate that there is a trade-off between computational load and reasoning accuracy when VLM analyzes a large number of high-resolution images. If we input all image sequences of $\mathcal{S}$ in a single round of dialogue, it may overload the VLM, degrade performance, or even cause response timeouts. Therefore, we slice the image sequences $\mathcal{S}$ into batches, allowing VLM to optimize the set of proposals over rounds, thus achieving precise localization of the target object.

The details of the mechanism are illustrated in Algorithm LABEL:algo:vlm-predict. In this algorithm, the variable $\mathcal{Q}$ is defined to dynamically maintain the image sequences that are to be analyzed. Before the iteration process begins, $\mathcal{Q}$ will be sliced into batches $\mathcal{B}=\{B_{1},B_{2},\dots,B_{q}\}$ with a maximum batch size $L$. Subsequently, each batch is combined with the textual description $T$ to form the input prompt for the VLM. If the VLM identifies a unique image sequence $S^{*}\in B_{i}$ that best matches $T$, $S^{*}$ will be reinserted to $\mathcal{Q}$; otherwise, $B_{i}$ will be discarded.

After each reasoning step, the algorithm checks the state of $\mathcal{Q}$. The iteration will stop only when $\lvert\mathcal{Q}\rvert$ is less than or equal to one. Finally, if only one image sequence remains in $\mathcal{Q}$, its corresponding index is selected as the final result. This mechanism circumvents the limitations of VLM in long-sequence reasoning by gradually reducing the search space.

For 3D localization, the bounding box of selected proposal can be retrieved from the Object Profile Table ($\mathcal{OPT}$), completing the end-to-end 3DVG pipeline.

4. Experiments

4.1. Experimental Settings

Datasets.

We evaluate our framework on two established 3D visual grounding benchmarks: ScanRefer[scanrefer] and Nr3D[referit3d]. ScanRefer, constructed from ScanNet, comprises 51,583 natural language descriptions paired with 11,046 objects across 800 indoor scenes, emphasizing fine-grained alignment between linguistic expressions and 3D object localization through attribute and spatial-relation reasoning. The dataset is divided into Unique and Multiple subsets: Unique targets objects uniquely identifiable by intrinsic attributes, while Multiple requires disambiguation among multiple instances of the same category using spatial relations. Nr3D contains 41,503 descriptions for 7,189 objects across 1,448 scenes, prioritizing viewpoint-dependent language diversity in real-world contexts. Its evaluation categorizes references into Easy and Hard subsets based on the necessity of implicit contextual reasoning, as well as View-Dependent and View-Independent subsets to assess robustness to observer perspectives.

Implementation Details.

Experiments were conducted on a TITAN RTX GPU using Doubao-1.5-pro[volcengine2025] as the LLM for semantic-guided proposal selection and Doubao-1.5-vision-pro[volcengine2025] as the VLM for multimodal reasoning. We employ Mask3D [mask3d] as the 3D semantic segmentation network, applying a confidence threshold of 0.2 to filter low-quality segmentation masks. Semantic matching is performed using the CLIP-ViT-Base-Patch16 text encoder [radford2021learning]. Scene image sequences are segmented into 20-frame intervals for viewpoint sampling, with each candidate object projected across five optimized viewpoints validated through depth consistency checks (visibility threshold is 0.25). We set the bounding region expansion parameter $\alpha$ to 0.25. The VLM processes candidates iteratively through an iterative reasoning mechanism with a batch size limit of four. Due to the high computational cost of VLM-based models, we follow the standardized protocol of VLM-Grounder [vlmgrounder] for fair evaluation and reproducibility, testing on 250 validation samples per benchmark dataset, matching previous evaluation conditions.

4.2. Quantitative Results

ScanRefer.

SeqVLM establishes a new state-of-the-art in zero-shot 3D visual grounding, as demonstrated by the comprehensive comparisons in Table LABEL:tab:scanrefer_result, while maintaining competitive performance relative to fully-supervised methods. Specifically, SeqVLM achieves 55.6% Acc@0.25 and 49.6% Acc@0.5 on the overall benchmark, outperforming previous state-of-the-art zero-shot methods, VLM-Grounder[vlmgrounder] and SeeGround[li2024seeground], by +4.0% and +10.2%, respectively. In single-object scenarios (Unique subset), SeqVLM surpasses prior approaches with gains of 1.6% and 3.8% in Acc@0.25 and Acc@0.5, respectively, highlighting its superior localization accuracy. For multi-object scenarios (Multiple subset), while Acc@0.25 marginally lags behind VLM-Grounder, our method achieves the highest Acc@0.5 performance (+7.8%), demonstrating enhanced fine-grained localization capabilities. Notably, the framework exhibits competitive results against fully-supervised baselines while maintaining strong generalization across diverse 3D visual grounding tasks. These results validate the effectiveness of the proposed method and underscore its potential for practical applications requiring robust 3D spatial reasoning.

Nr3D.

As shown in Table  LABEL:tab:nr3d_result, SeqVLM attains an overall accuracy of 53.2% on the Nr3D dataset, exceeding prior state-of-the-art zero-shot methods by 5.2%. Our method exhibits robust performance across varying scenario complexities, achieving 58.1% (+2.9%) and 47.4% (+7.9%) accuracy in the ”Easy” and ”Hard” subsets, respectively. Equally notable is its performance in view-dependent and view-independent scenarios, with accuracies of 51.0% (+5.2%) and 54.5% (+5.1%), illustrating its effective handling of perspective-dependent variations.

4.3. Qualitative Results

Figure 3 presents the qualitative 3D visual grounding results of ZSVG3D[zs3dvg], SeeGround [li2024seeground], and SeqVLM on the ScanRefer [scanrefer] dataset. Cases (a–c) demonstrate the superior localization capability of our model, even in complex environments with challenging textual descriptions. For instance, in case (a), our model accurately identifies the target chair despite multiple chairs being present in the scene. In contrast, both baseline methods exhibit lower localization accuracy, with ZSVG3D occasionally misclassifying object categories (e.g., identifying bins instead of cabinets in case (c)). These qualitative results underscore the efficacy of our approach in enhancing grounding precision and robustness.

Figure 4 contrasts the visual inputs provided to VLM by See-Ground [li2024seeground] and our method. The ground-truth (a) depicts “the gray padded chair on rollers. it is the only rolling chair at the table.” However, SeeGround’s single-view rendering (b) occludes these distinguishing features due to viewpoint constraints, resulting in misidentification. In contrast, our multi-view sequence (c) integrates complementary perspectives, resolving occlusions and clearly revealing the wheels. This visual evidence directly accounts for the accurate localization of our method, demonstrating that multi-view fusion mitigates single-view ambiguities inherent in real-world 3D grounding tasks.

4.4. Ablation Studies

Component Ablation.

To evaluate the contribution of core components in our framework, comprehensive ablation studies are conducted on the ScanRefer benchmark as shown in Table  LABEL:tab:ablation_module. The baseline model achieves merely 2.4% accuracy under the Acc@0.5 metric, highlighting the inherent complexity of zero-shot 3D visual grounding. Introducing the Proposal Selection Module significantly improves performance to 43.2% by filtering semantically relevant candidates through text-driven category alignment, whereas standalone implementations of the Multi-View Projection technique or Iterative Reasoning mechanism yield limited gains of 2.0% and 10.0% respectively. The complete framework integrating all components attains 49.6% accuracy, demonstrating substantial synergistic effects through the combination of semantic-aware multi-view 2D visual representations and iterative candidate refinement. This progression underscores the indispensability of each module, particularly the Proposal Selection Module which contributes the most critical performance leap by bridging 3D segmentation with language semantics.

VLM Selection.

We evaluate the influence of different VLMs on grounding performance and cost. As shown in Table LABEL:tab:ablation_vlm, GPT-4[chatgpt2024] achieves 44.0 Acc@0.5 with moderate token usage (1,684k) and cost (2.2$), while Qwen-vl-max[qwen2025] improves accuracy to 46.0 at a lower cost (0.8$). Doubao-1.5-vision-pro achieves the highest accuracy (49.6 Acc@0.5) with higher token usage (7,097k), indicating superior visual-text alignment. Given the importance of precision in 3DVG, we choose Doubao as the default VLM, favoring its accuracy despite increased computational cost.

Cross-Method Comparison.

To decouple the impact of framework design from VLM selection, we conduct cross-method comparisons under controlled VLM settings in Table  LABEL:tab:ablation_method. When utilizing GPT-4, our SeqVLM achieves 44.0 Acc@0.5, outperforming VLM-Grounder’s 32.8 by 11.2 points, confirming fundamental architectural advantages in multimodal fusion and geometric reasoning. Simultaneously, under the Doubao VLM environment, SeqVLM attains 49.6 Acc@0.5, outperforming ZSVG3D’s 29.2 and SeeGround’s 40.0 by 20.4 and 9.6 points, respectively. These consistent superiority patterns across heterogeneous VLM backends demonstrate that our framework’s innovations in proposal selection, proposal-guided multi-view projection, and iterative reasoning constitute the primary driver of performance gains, rather than mere VLM capability differences. The results establish SeqVLM as a generalized architectural advancement for zero-shot 3D grounding, transferable across modern VLMs.

Hyper-parameter Analysis

Our ablation reveals that both the VLM capacity threshold $L$ and multi-view frame number $n_{\mathit{frame}}$ exhibit inverse U-shaped effects on accuracy, peaking at $L=4$ and $n_{\mathit{frame}}=5$ (Figures 5-6). Smaller capacity threshold ($L=2$) limit cross-candidate contrast through insufficient batch diversity, while larger capacity threshold ($L\geq 6$) overload the VLM’s multimodal reasoning. Similarly for, fewer views ($n_{\mathit{frame}}\leq 4$) restrict spatial disambiguation and feature richness, whereas excessive views ($n_{\mathit{frame}}\geq 6$) introduce redundant or occluded projections as cross-modal noise. The chosen setting ($L=4$, $n_{\mathit{frame}}=5$) provides the best trade-off between information completeness and computational load, maximizing alignment within VLM capacity.

5. Conclusion

In this paper, we propose SeqVLM, a novel framework for zero-shot 3D visual grounding that leverages multi-view real-world scene images. Our approach first employs a Proposal Selection Module to eliminate semantically irrelevant proposals, improving both reasoning accuracy and efficiency. Then, we introduce a proposal-guided multi-view projection strategy to mitigate projection misalignment and enrich contextual information across different viewpoints. To enhance the reasoning capabilities of the VLM, we further design an iterative reasoning mechanism. Evaluations on ScanRefer and Nr3D benchmarks demonstrate that SeqVLM achieves state-of-the-art performance in zero-shot 3D visual grounding, matching the accuracy of supervised approaches while requiring no task-specific training.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62176224, Grant 62176092, Grant 62222602, and Grant 62306165; in part by Science and Technology on Sonar Laboratory under grant 2024-JCJQ-LB-32/07 and in part by China Academy of Railway Sciences under Grant 2023Y1357.