"Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration

TL;DR

This paper proposes a new approach for creating ecologically valid benchmarks for situated collaboration by generating questions interactively during user interactions with AI systems, aiming to better reflect real-world scenarios.

Contribution

It introduces an interactive, system-driven method for generating benchmark questions, contrasting with traditional static datasets, to improve assessment of multimodal models in real-world contexts.

Findings

Questions differ significantly from existing benchmarks in form and content.

The approach reveals new real-world challenge problems.

Highlights the importance of context-aware evaluation.

Abstract

We report initial work towards constructing ecologically valid benchmarks to assess the capabilities of large multimodal models for engaging in situated collaboration. In contrast to existing benchmarks, in which question-answer pairs are generated post hoc over preexisting or synthetic datasets via templates, human annotators, or large language models (LLMs), we propose and investigate an interactive system-driven approach, where the questions are generated by users in context, during their interactions with an end-to-end situated AI system. We illustrate how the questions that arise are different in form and content from questions typically found in existing embodied question answering (EQA) benchmarks and discuss new real-world challenge problems brought to the fore.