ReLI: A Language-Agnostic Approach to Human-Robot Interaction

TL;DR

ReLI is a novel language-agnostic framework that enables autonomous agents to understand and execute human instructions across multiple languages through natural conversation and reasoning, advancing inclusive human-robot interaction.

Contribution

ReLI introduces a cross-lingual, natural language-based approach for human-robot interaction, grounded in large-scale pre-trained models, capable of handling diverse languages and tasks.

Findings

Achieved over 90% accuracy in cross-lingual instruction parsing

Demonstrated robustness on 140 languages with 70K+ conversations

Effective in zero- and few-shot spatial navigation and query tasks

Abstract

Adapting autonomous agents for real-world industrial, domestic, and other daily tasks is currently gaining momentum. However, in global or cross-lingual application contexts, ensuring effective interaction with the environment and executing unrestricted human-specified tasks regardless of the language remains an unsolved problem. To address this, we propose ReLI, a language-agnostic approach that enables autonomous agents to converse naturally, semantically reason about their environment, and perform downstream tasks, regardless of the task instruction's modality or linguistic origin. First, we ground large-scale pre-trained foundation models and transform them into language-to-action models that can directly provide common-sense reasoning and high-level robot control through natural, free-flow conversational interactions. Further, we perform cross-lingual adaptation of the models to ensure that ReLI generalises across the global languages. To demonstrate ReLI's robustness, we conducted extensive experiments on various short- and long-horizon tasks, including zero- and few-shot spatial navigation, scene information retrieval, and query-oriented tasks. We benchmarked the performance on $140$ languages involving $70K+$ multi-turn conversations. On average, ReLI achieved over $90\%\pm0.2$ accuracy in cross-lingual instruction parsing and task execution success. These results demonstrate its potential to advance natural human-agent interaction in the real world while championing inclusive and linguistic diversity. Demos and resources will be public at: https://linusnep.github.io/ReLI/.