Semantic Skill Grounding for Embodied Instruction-Following in Cross-Domain Environments

TL;DR

This paper introduces SemGro, a hierarchical semantic skill grounding framework that improves embodied instruction-following across diverse domains by leveraging language models' reasoning and multi-modal capabilities.

Contribution

SemGro is a novel hierarchical framework that decomposes and grounds semantic skills in target domains, enhancing cross-domain embodied instruction-following.

Findings

Outperforms baselines in 300 cross-domain scenarios

Effective hierarchical skill decomposition and grounding

Leverages language models for reasoning and feasibility assessment

Abstract

In embodied instruction-following (EIF), the integration of pretrained language models (LMs) as task planners emerges as a significant branch, where tasks are planned at the skill level by prompting LMs with pretrained skills and user instructions. However, grounding these pretrained skills in different domains remains challenging due to their intricate entanglement with the domain-specific knowledge. To address this challenge, we present a semantic skill grounding (SemGro) framework that leverages the hierarchical nature of semantic skills. SemGro recognizes the broad spectrum of these skills, ranging from short-horizon low-semantic skills that are universally applicable across domains to long-horizon rich-semantic skills that are highly specialized and tailored for particular domains. The framework employs an iterative skill decomposition approach, starting from the higher levels of semantic skill hierarchy and then moving downwards, so as to ground each planned skill to an executable level within the target domain. To do so, we use the reasoning capabilities of LMs for composing and decomposing semantic skills, as well as their multi-modal extension for assessing the skill feasibility in the target domain. Our experiments in the VirtualHome benchmark show the efficacy of SemGro in 300 cross-domain EIF scenarios.