VCA: Vision-Click-Action Framework for Precise Manipulation of Segmented Objects in Target Ambiguous Environments

TL;DR

VCA introduces a click-based visual interaction framework for robotic manipulation, replacing language commands with direct visual selection to improve accuracy and reduce ambiguity in object targeting.

Contribution

The paper presents VCA, a novel framework that leverages visual clicks and pretrained segmentation models to enhance robotic object manipulation in ambiguous environments.

Findings

Effective instance-level manipulation demonstrated

Reduces interpretation errors and cognitive load

Scalable and practical for real-world applications

Abstract

The reliance on language in Vision-Language-Action (VLA) models introduces ambiguity, cognitive overhead, and difficulties in precise object identification and sequential task execution, particularly in environments with multiple visually similar objects. To address these limitations, we propose Vision-Click-Action (VCA), a framework that replaces verbose textual commands with direct, click-based visual interaction using pretrained segmentation models. By allowing operators to specify target objects clearly through visual selection in the robot's 2D camera view, VCA reduces interpretation errors, lowers cognitive load, and provides a practical and scalable alternative to language-driven interfaces for real-world robotic manipulation. Experimental results validate that the proposed VCA framework achieves effective instance-level manipulation of specified target objects. Experiment videos are available at https://robrosinc.github.io/vca/.