Towards Natural Language Environment: Understanding Seamless Natural-Language-Based Human-Multi-Robot Interactions

TL;DR

This paper explores the design space of Natural Language Environments (NLEs) for seamless human-multi-robot interactions using role-playing VR studies, highlighting key design tensions and opportunities.

Contribution

It introduces the concept of NLEs, synthesizes prior work, and investigates human conceptualization and coordination in multi-robot environments through VR role-playing.

Findings

Refined design space for NLEs based on user studies

Identified tensions in task coordination and robot autonomy

Derived design implications for future NLE development

Abstract

As multiple robots are expected to coexist in future households, natural language is increasingly envisioned as a primary medium for human-robot and robot-robot communication. This paper introduces the concept of a Natural Language Environment (NLE), defined as an interaction space in which humans and multiple heterogeneous robots coordinate primarily through natural language. Rather than proposing a deployable system, this work aims to explore the design space of such environments. We first synthesize prior work on language-based human-robot interaction to derive a preliminary design space for NLEs. We then conduct a role-playing study in virtual reality to investigate how people conceptualize, negotiate, and coordinate human-multi-robot interactions within this imagined environment. Based on qualitative and quantitative analysis, we refine the preliminary design space and derive design implications that highlight key tensions and opportunities around task coordination dominance, robot autonomy, and robot personality in Natural Language Environments.