Lifelong Language-Conditioned Robotic Manipulation Learning

TL;DR

SkillsCrafter is a continual learning framework for robotic manipulation that reduces forgetting of old skills and efficiently learns new skills by leveraging shared semantic subspaces and skill aggregation.

Contribution

It introduces a novel continual learning approach with semantic subspace projection and skill aggregation to improve lifelong manipulation learning.

Findings

Outperforms existing methods in retaining old skills.

Effectively learns new manipulation skills with minimal forgetting.

Demonstrates superior generalization to unknown skills.

Abstract

Traditional language-conditioned manipulation agent sequential adaptation to new manipulation skills leads to catastrophic forgetting of old skills, limiting dynamic scene practical deployment. In this paper, we propose SkillsCrafter, a novel robotic manipulation framework designed to continually learn multiple skills while reducing catastrophic forgetting of old skills. Specifically, we propose a Manipulation Skills Adaptation to retain the old skills knowledge while inheriting the shared knowledge between new and old skills to facilitate learning of new skills. Meanwhile, we perform the singular value decomposition on the diverse skill instructions to obtain common skill semantic subspace projection matrices, thereby recording the essential semantic space of skills. To achieve forget-less and generalization manipulation, we propose a Skills Specialization Aggregation to compute inter-skills similarity in skill semantic subspaces, achieving aggregation of the previously learned skill knowledge for any new or unknown skill. Extensive experiments demonstrate the effectiveness and superiority of our proposed SkillsCrafter.