Efficient Morphology-Aware Policy Transfer to New Embodiments

TL;DR

This paper introduces a method combining morphology-aware pretraining with parameter efficient finetuning to improve zero-shot transfer and reduce data requirements for policy adaptation in robotics.

Contribution

It demonstrates that PEFT techniques, when combined with morphology-aware pretraining, significantly enhance policy transfer efficiency with minimal parameter tuning.

Findings

PEFT techniques reduce sample complexity for policy adaptation.

Tuning less than 1% of parameters improves zero-shot policy performance.

Combining pretraining with PEFT outperforms end-to-end finetuning from scratch.

Abstract

Morphology-aware policy learning is a means of enhancing policy sample efficiency by aggregating data from multiple agents. These types of policies have previously been shown to help generalize over dynamic, kinematic, and limb configuration variations between agent morphologies. Unfortunately, these policies still have sub-optimal zero-shot performance compared to end-to-end finetuning on morphologies at deployment. This limitation has ramifications in practical applications such as robotics because further data collection to perform end-to-end finetuning can be computationally expensive. In this work, we investigate combining morphology-aware pretraining with parameter efficient finetuning (PEFT) techniques to help reduce the learnable parameters necessary to specialize a morphology-aware policy to a target embodiment. We compare directly tuning sub-sets of model weights, input learnable adapters, and prefix tuning techniques for online finetuning. Our analysis reveals that PEFT techniques in conjunction with policy pre-training generally help reduce the number of samples to necessary to improve a policy compared to training models end-to-end from scratch. We further find that tuning as few as less than 1% of total parameters will improve policy performance compared the zero-shot performance of the base pretrained a policy.