AdaptBot: Combining LLM with Knowledge Graphs and Human Input for Generic-to-Specific Task Decomposition and Knowledge Refinement

TL;DR

AdaptBot integrates Large Language Models, Knowledge Graphs, and human input to enable robots to quickly adapt to new tasks in domains like cooking and cleaning, improving performance over LLM-only approaches.

Contribution

This work introduces a novel framework combining LLMs, knowledge graphs, and human input for efficient task decomposition and knowledge refinement in embodied agents.

Findings

Performance improved by leveraging KG and human input

Effective in simulation-based cooking and cleaning tasks

Significant gains over LLM-only methods

Abstract

An embodied agent assisting humans is often asked to complete new tasks, and there may not be sufficient time or labeled examples to train the agent to perform these new tasks. Large Language Models (LLMs) trained on considerable knowledge across many domains can be used to predict a sequence of abstract actions for completing such tasks, although the agent may not be able to execute this sequence due to task-, agent-, or domain-specific constraints. Our framework addresses these challenges by leveraging the generic predictions provided by LLM and the prior domain knowledge encoded in a Knowledge Graph (KG), enabling an agent to quickly adapt to new tasks. The robot also solicits and uses human input as needed to refine its existing knowledge. Based on experimental evaluation in the context of cooking and cleaning tasks in simulation domains, we demonstrate that the interplay between LLM, KG, and human input leads to substantial performance gains compared with just using the LLM. Project website{\S}: https://sssshivvvv.github.io/adaptbot/