Sample-Efficient Behavior Cloning Using General Domain Knowledge

TL;DR

This paper introduces KIM, a method that leverages large language models to incorporate expert domain knowledge into behavior cloning, significantly improving sample efficiency and robustness in decision-making tasks.

Contribution

The paper proposes a novel approach to embed domain knowledge into policy structures using language models, enhancing learning efficiency and generalization in behavior cloning.

Findings

Learns with as few as 5 demonstrations

Outperforms baseline models without domain knowledge

Shows robustness to action noise

Abstract

Behavior cloning has shown success in many sequential decision-making tasks by learning from expert demonstrations, yet they can be very sample inefficient and fail to generalize to unseen scenarios. One approach to these problems is to introduce general domain knowledge, such that the policy can focus on the essential features and may generalize to unseen states by applying that knowledge. Although this knowledge is easy to acquire from the experts, it is hard to be combined with learning from individual examples due to the lack of semantic structure in neural networks and the time-consuming nature of feature engineering. To enable learning from both general knowledge and specific demonstration trajectories, we use a large language model's coding capability to instantiate a policy structure based on expert domain knowledge expressed in natural language and tune the parameters in the policy with demonstrations. We name this approach the Knowledge Informed Model (KIM) as the structure reflects the semantics of expert knowledge. In our experiments with lunar lander and car racing tasks, our approach learns to solve the tasks with as few as 5 demonstrations and is robust to action noise, outperforming the baseline model without domain knowledge. This indicates that with the help of large language models, we can incorporate domain knowledge into the structure of the policy, increasing sample efficiency for behavior cloning.