Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions

TL;DR

This paper introduces a knowledge-driven imitation learning framework that uses external semantic knowledge to improve generalization in robot manipulation, achieving better performance with fewer demonstrations across diverse conditions.

Contribution

It proposes a novel semantic keypoint graph and a template-matching algorithm to enhance object representation and generalization in imitation learning.

Findings

Outperforms image-based diffusion policies with fewer demonstrations

Demonstrates robustness across different objects, backgrounds, and lighting

Achieves superior performance on real-world robotic tasks

Abstract

Imitation learning has emerged as a powerful paradigm in robot manipulation, yet its generalization capability remains constrained by object-specific dependencies in limited expert demonstrations. To address this challenge, we propose knowledge-driven imitation learning, a framework that leverages external structural semantic knowledge to abstract object representations within the same category. We introduce a novel semantic keypoint graph as a knowledge template and develop a coarse-to-fine template-matching algorithm that optimizes both structural consistency and semantic similarity. Evaluated on three real-world robotic manipulation tasks, our method achieves superior performance, surpassing image-based diffusion policies with only one-quarter of the expert demonstrations. Extensive experiments further demonstrate its robustness across novel objects, backgrounds, and lighting conditions. This work pioneers a knowledge-driven approach to data-efficient robotic learning in real-world settings. Code and more materials are available on https://knowledge-driven.github.io/.