Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning

TL;DR

This paper presents CLIPS, a Bayesian agent architecture that uses large language models to interpret ambiguous instructions and assist humans in goal achievement through pragmatic, context-aware planning in cooperative tasks.

Contribution

It introduces a novel cooperative language-guided inverse planning framework that effectively models human intentions and improves assistive behavior in ambiguous instruction scenarios.

Findings

CLIPS outperforms GPT-4V and other baselines in accuracy and helpfulness.

The approach closely matches human judgments in cooperative planning tasks.

Demonstrates effective goal inference and assistance in two domain environments.

Abstract

People often give instructions whose meaning is ambiguous without further context, expecting that their actions or goals will disambiguate their intentions. How can we build assistive agents that follow such instructions in a flexible, context-sensitive manner? This paper introduces cooperative language-guided inverse plan search (CLIPS), a Bayesian agent architecture for pragmatic instruction following and goal assistance. Our agent assists a human by modeling them as a cooperative planner who communicates joint plans to the assistant, then performs multimodal Bayesian inference over the human's goal from actions and language, using large language models (LLMs) to evaluate the likelihood of an instruction given a hypothesized plan. Given this posterior, our assistant acts to minimize expected goal achievement cost, enabling it to pragmatically follow ambiguous instructions and provide effective assistance even when uncertain about the goal. We evaluate these capabilities in two cooperative planning domains (Doors, Keys & Gems and VirtualHome), finding that CLIPS significantly outperforms GPT-4V, LLM-based literal instruction following and unimodal inverse planning in both accuracy and helpfulness, while closely matching the inferences and assistive judgments provided by human raters.