Open-Ended Goal Inference through Actions and Language for Human-Robot Collaboration

TL;DR

This paper introduces BALI, a goal inference method for human-robot collaboration that combines language and actions, asks clarifying questions strategically, and improves goal prediction stability in complex tasks.

Contribution

BALI is a novel approach that integrates language and action cues for goal inference, allowing robots to handle unbounded and novel goals in collaborative tasks.

Findings

BALI achieves more stable goal predictions than baselines.

BALI makes significantly fewer mistakes in goal inference.

BALI effectively combines language and action cues for improved inference.

Abstract

To collaborate with humans, robots must infer goals that are often ambiguous, difficult to articulate, or not drawn from a fixed set. Prior approaches restrict inference to a predefined goal set, rely only on observed actions, or depend exclusively on explicit instructions, making them brittle in real-world interactions. We present BALI (Bidirectional Action-Language Inference) for goal prediction, a method that integrates natural language preferences with observed human actions in a receding-horizon planning tree. BALI combines language and action cues from the human, asks clarifying questions only when the expected information gain from the answer outweighs the cost of interruption, and selects supportive actions that align with inferred goals. We evaluate the approach in collaborative cooking tasks, where goals may be novel to the robot and unbounded. Compared to baselines, BALI yields more stable goal predictions and significantly fewer mistakes.