Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraints

TL;DR

This paper introduces a novel offline algorithm that enhances diversity in skill learning by leveraging Van der Waals forces and successor features, improving stability and skill recall in robotic tasks.

Contribution

The proposed method combines VdW-based diversity maximization with pre-trained functional reward encoding, eliminating the need for skill discriminators and enabling zero-shot skill recall.

Findings

Effective in generating diverse robotic skills in simulation

Improves training stability and efficiency

Expands skill repertoire with zero-shot recall

Abstract

While many algorithms for diversity maximization under imitation constraints are online in nature, many applications require offline algorithms without environment interactions. Tackling this problem in the offline setting, however, presents significant challenges that require non-trivial, multi-stage optimization processes with non-stationary rewards. In this work, we present a novel offline algorithm that enhances diversity using an objective based on Van der Waals (VdW) force and successor features, and eliminates the need to learn a previously used skill discriminator. Moreover, by conditioning the value function and policy on a pre-trained Functional Reward Encoding (FRE), our method allows for better handling of non-stationary rewards and provides zero-shot recall of all skills encountered during training, significantly expanding the set of skills learned in prior work. Consequently, our algorithm benefits from receiving a consistently strong diversity signal (VdW), and enjoys more stable and efficient training. We demonstrate the effectiveness of our method in generating diverse skills for two robotic tasks in simulation: locomotion of a quadruped and local navigation with obstacle traversal.