PointArena: Probing Multimodal Grounding Through Language-Guided Pointing

TL;DR

PointArena introduces a comprehensive platform with datasets, benchmarks, and real-world evaluation tools to advance multimodal grounding through language-guided pointing, highlighting the importance of specialized training for improved performance.

Contribution

We present PointArena, a new multi-component platform for evaluating and comparing multimodal pointing abilities across reasoning and real-world tasks.

Findings

Molmo-72B outperforms other models in pointing tasks

Supervised training on pointing tasks improves model performance

Strong correlations found across evaluation stages

Abstract

Pointing serves as a fundamental and intuitive mechanism for grounding language within visual contexts, with applications spanning robotics, assistive technologies, and interactive AI systems. While recent multimodal models have started to support pointing capabilities, existing benchmarks typically focus only on referential object localization tasks. We introduce PointArena, a comprehensive platform for evaluating multimodal pointing across diverse reasoning scenarios. PointArena comprises three components: (1) Point-Bench, a curated dataset containing approximately 1,000 pointing tasks across five reasoning categories; (2) Point-Battle, an interactive, web-based arena facilitating blind, pairwise model comparisons, which has already gathered over 4,500 anonymized votes; and (3) Point-Act, a real-world robotic manipulation system allowing users to directly evaluate multimodal model pointing capabilities in practical settings. We conducted extensive evaluations of both state-of-the-art open-source and proprietary multimodal models. Results indicate that Molmo-72B consistently outperforms other models, though proprietary models increasingly demonstrate comparable performance. Additionally, we find that supervised training specifically targeting pointing tasks significantly enhances model performance. Across our multi-stage evaluation pipeline, we also observe strong correlations, underscoring the critical role of precise pointing capabilities in enabling multimodal models to effectively bridge abstract reasoning with concrete, real-world actions. Project page: https://pointarena.github.io/