Continual Robot Skill and Task Learning via Dialogue

TL;DR

This paper introduces a framework for continual robot skill learning through dialog interactions, featuring a novel visual-motor control policy that improves learning efficiency and effectiveness in simulation and human-robot interaction scenarios.

Contribution

It presents ACT-LoRA, a new visual-motor policy enabling continual skill learning from few demonstrations, and a dialog-based framework for human-guided robot skill acquisition.

Findings

ACT-LoRA outperforms GMM-LoRA by over 300% on simulation benchmarks.

The dialog framework enables 100% success in teaching cooking skills in human studies.

Robots can learn new skills efficiently with minimal demonstrations.

Abstract

Interactive robot learning is a challenging problem as the robot is present with human users who expect the robot to learn novel skills to solve novel tasks perpetually with sample efficiency. In this work we present a framework for robots to continually learn tasks and visuo-motor skills and query for novel skills via dialog interactions with human users. Our robot agent maintains a skill library, and uses an existing LLM to perform grounded dialog interactions to query unknown skills from real human users. We developed a novel visual-motor control policy Action Chunking Transformer with Low Rank Adaptation (ACT-LoRA) that can continually learn novel skills using only a few demonstrations which is critical in human-robot interaction scenarios. The paper has twin goals: Firstly to demonstrate better continual learning in simulation; and secondly, to demonstrate the use of our dialog based learning framework in a realistic human-robot interaction use case. Our ACT-LoRA policy consistently outperforms a GMM-LoRA baseline on multiple continual learning simulation benchmarks by achieving > 300% improvements on novel skills, while achieving comparable performance in existing skills. Moreover, with our IRB approved human-subjects study we demonstrate that our dialog based continual learning framework allows users to teach robots cooking skills successfully (100%) while spending a higher ratio of time on finishing an auxiliary distraction tasks in the test phase of the study compared to a non-learning language based agent (p < 0.001).