Quantification before Selection: Active Dynamics Preference for Robust Reinforcement Learning

TL;DR

This paper introduces Active Dynamics Preference (ADP), a method that actively samples system parameters during training to improve robustness of policies in reinforcement learning, outperforming traditional domain randomization.

Contribution

The paper proposes ADP, a novel active sampling strategy that dynamically selects system parameters based on informativeness and density to enhance policy robustness.

Findings

ADP outperforms baseline methods in robotic locomotion tasks.

Active sampling stabilizes training and prevents over-conservativeness.

Robustness to environment discrepancies is significantly improved.

Abstract

Training a robust policy is critical for policy deployment in real-world systems or dealing with unknown dynamics mismatch in different dynamic systems. Domain Randomization~(DR) is a simple and elegant approach that trains a conservative policy to counter different dynamic systems without expert knowledge about the target system parameters. However, existing works reveal that the policy trained through DR tends to be over-conservative and performs poorly in target domains. Our key insight is that dynamic systems with different parameters provide different levels of difficulty for the policy, and the difficulty of behaving well in a system is constantly changing due to the evolution of the policy. If we can actively sample the systems with proper difficulty for the policy on the fly, it will stabilize the training process and prevent the policy from becoming over-conservative or over-optimistic. To operationalize this idea, we introduce Active Dynamics Preference~(ADP), which quantifies the informativeness and density of sampled system parameters. ADP actively selects system parameters with high informativeness and low density. We validate our approach in four robotic locomotion tasks with various discrepancies between the training and testing environments. Extensive results demonstrate that our approach has superior robustness for system inconsistency compared to several baselines.