Situated Instruction Following
So Yeon Min, Xavi Puig, Devendra Singh Chaplot, Tsung-Yen Yang,, Akshara Rai, Priyam Parashar, Ruslan Salakhutdinov, Yonatan Bisk, Roozbeh, Mottaghi

TL;DR
This paper emphasizes the importance of understanding situated instructions in robotic assistants, highlighting the limitations of current models in grasping the contextual and evolving nature of human communication within physical environments.
Contribution
It introduces the concept of situated instruction following that incorporates real-world context, human actions, and environmental cues into instruction understanding.
Findings
State-of-the-art EIF models lack holistic understanding of situated human intention.
Situated instructions are often ambiguously specified and temporally evolving.
Understanding requires integrating past actions and future behaviors of humans.
Abstract
Language is never spoken in a vacuum. It is expressed, comprehended, and contextualized within the holistic backdrop of the speaker's history, actions, and environment. Since humans are used to communicating efficiently with situated language, the practicality of robotic assistants hinge on their ability to understand and act upon implicit and situated instructions. In traditional instruction following paradigms, the agent acts alone in an empty house, leading to language use that is both simplified and artificially "complete." In contrast, we propose situated instruction following, which embraces the inherent underspecification and ambiguity of real-world communication with the physical presence of a human speaker. The meaning of situated instructions naturally unfold through the past actions and the expected future behaviors of the human involved. Specifically, within our settings we…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSocial Robot Interaction and HRI · Multimodal Machine Learning Applications · Language and cultural evolution
