Flow-based Domain Randomization for Learning and Sequencing Robotic Skills

TL;DR

This paper introduces a method that automatically discovers flexible environment sampling distributions using flow-based models, enhancing robustness and out-of-distribution detection in robotic control policies trained via domain randomization.

Contribution

It proposes a novel entropy-regularized approach with normalizing flows for automatic environment distribution discovery, outperforming simpler methods in robustness and generalization.

Findings

Flow-based sampling distributions improve robustness in simulation and real-world robotics.

The method outperforms existing approaches in six simulated and one real-world domain.

Learned distributions enable effective out-of-distribution detection in manipulation planning.

Abstract

Domain randomization in reinforcement learning is an established technique for increasing the robustness of control policies trained in simulation. By randomizing environment properties during training, the learned policy can become robust to uncertainties along the randomized dimensions. While the environment distribution is typically specified by hand, in this paper we investigate automatically discovering a sampling distribution via entropy-regularized reward maximization of a normalizing-flow-based neural sampling distribution. We show that this architecture is more flexible and provides greater robustness than existing approaches that learn simpler, parameterized sampling distributions, as demonstrated in six simulated and one real-world robotics domain. Lastly, we explore how these learned sampling distributions, combined with a privileged value function, can be used for out-of-distribution detection in an uncertainty-aware multi-step manipulation planner.