Point What You Mean: Visually Grounded Instruction Policy

TL;DR

This paper introduces Point-VLA, a visual grounding policy that enhances vision-language-action models with explicit visual cues, significantly improving object referring accuracy in cluttered and OOD scenes.

Contribution

The study presents a plug-and-play visual grounding policy and an automatic data annotation pipeline, enabling better object-level grounding and generalization in embodied control tasks.

Findings

Point-VLA outperforms text-only models in referring tasks.

It achieves robust generalization to unseen objects.

It effectively resolves referential ambiguity in complex scenes.

Abstract

Vision-Language-Action (VLA) models align vision and language with embodied control, but their object referring ability remains limited when relying solely on text prompt, especially in cluttered or out-of-distribution (OOD) scenes. In this study, we introduce the Point-VLA, a plug-and-play policy that augments language instructions with explicit visual cues (e.g., bounding boxes) to resolve referential ambiguity and enable precise object-level grounding. To efficiently scale visually grounded datasets, we further develop an automatic data annotation pipeline requiring minimal human effort. We evaluate Point-VLA on diverse real-world referring tasks and observe consistently stronger performance than text-only instruction VLAs, particularly in cluttered or unseen-object scenarios, with robust generalization. These results demonstrate that Point-VLA effectively resolves object referring ambiguity through pixel-level visual grounding, achieving more generalizable embodied control.