Adapt as You Say: Online Interactive Bimanual Skill Adaptation via Human Language Feedback

TL;DR

This paper introduces BiSAIL, a framework for zero-shot online adaptation of bimanual robot skills using interactive language feedback, enabling flexible, human-in-the-loop customization in real-world tasks.

Contribution

BiSAIL is a novel hierarchical approach that infers adaptation goals from multimodal feedback and modulates bimanual motions via diffusion, advancing robot adaptability and generalization.

Findings

BiSAIL outperforms existing methods in adaptability and generalization.

Demonstrated on six bimanual tasks across two platforms.

Enables non-expert users to customize robots through verbal corrections.

Abstract

Developing general-purpose robots capable of autonomously operating in human living environments requires the ability to adapt to continuously evolving task conditions. However, adapting high-dimensional coordinated bimanual skills to novel task variations at deployment remains a fundamental challenge. In this work, we present BiSAIL (Bimanual Skill Adaptation via Interactive Language), a novel framework that enables zero-shot online adaptation of offline-learned bimanual skills through interactive language feedback. The key idea of BiSAIL is to adopt a hierarchical reason-then-modulate paradigm, which first infers generalized adaptation objectives from multimodal task variations, and then adapts bimanual motions via diffusion modulation to achieve the inferred objectives. Extensive real-robot experiments across six bimanual tasks and two dual-arm platforms demonstrate that BiSAIL significantly outperforms existing methods in human-in-the-loop adaptability, task generalization and cross-embodiment scalability. This work enables the development of adaptive bimanual assistants that can be flexibly customized by non-expert users via intuitive verbal corrections. Experimental videos and code are available at https://rip4kobe.github.io/BiSAIL/.