TRACE: Textual Reasoning for Affordance Coordinate Extraction

TL;DR

This paper introduces TRACE, a method that enhances vision-language models for robotic affordance prediction by integrating textual reasoning, leading to improved accuracy, interpretability, and robustness in spatial understanding tasks.

Contribution

The paper presents a novel textual reasoning approach and a large-scale dataset that significantly improve VLM performance in affordance extraction for robotics.

Findings

Achieved 48.1% accuracy on W2P benchmark, a 9.6% improvement.

Model performance scales with reasoning data size.

Attention maps show interpretable, dynamic focus during reasoning.

Abstract

Vision-Language Models (VLMs) struggle to translate high-level instructions into the precise spatial affordances required for robotic manipulation. While visual Chain-of-Thought (CoT) methods exist, they are often computationally intensive. In this work, we introduce TRACE (Textual Reasoning for Affordance Coordinate Extraction), a novel methodology that integrates a textual Chain of Reasoning (CoR) into the affordance prediction process. We use this methodology to create the TRACE dataset, a large-scale collection created via an autonomous pipeline that pairs instructions with explicit textual rationales. By fine-tuning a VLM on this data, our model learns to externalize its spatial reasoning before acting. Our experiments show that our TRACE-tuned model achieves state-of-the-art performance, reaching 48.1% accuracy on the primary Where2Place (W2P) benchmark (a 9.6% relative improvement) and 55.0% on the more challenging W2P(h) subset. Crucially, an ablation study demonstrates that performance scales directly with the amount of reasoning data used, confirming the CoR's effectiveness. Furthermore, analysis of the model's attention maps reveals an interpretable reasoning process where focus shifts dynamically across reasoning steps. This work shows that training VLMs to generate a textual CoR is an effective and robust strategy for enhancing the precision, reliability, and interpretability of VLM-based robot control. Our dataset and code are available at https://github.com/jink-ucla/TRACE