Dialogue Object Search

TL;DR

This paper introduces the dialogue object search task, where robots communicate with humans via dialogue and video to locate objects in environments, advancing collaborative robot capabilities.

Contribution

It defines a new task combining dialogue and visual search, providing initial analysis and discussing future challenges for developing intelligent collaborative robots.

Findings

Collected pilot data demonstrating task feasibility

Analyzed examples of robot-human dialogue interactions

Outlined key challenges for future research in dialogue-based search

Abstract

We envision robots that can collaborate and communicate seamlessly with humans. It is necessary for such robots to decide both what to say and how to act, while interacting with humans. To this end, we introduce a new task, dialogue object search: A robot is tasked to search for a target object (e.g. fork) in a human environment (e.g., kitchen), while engaging in a "video call" with a remote human who has additional but inexact knowledge about the target's location. That is, the robot conducts speech-based dialogue with the human, while sharing the image from its mounted camera. This task is challenging at multiple levels, from data collection, algorithm and system development,to evaluation. Despite these challenges, we believe such a task blocks the path towards more intelligent and collaborative robots. In this extended abstract, we motivate and introduce the dialogue object search task and analyze examples collected from a pilot study. We then discuss our next steps and conclude with several challenges on which we hope to receive feedback.