Composable Learning with Sparse Kernel Representations

TL;DR

This paper introduces a reinforcement learning method that learns sparse kernel-based controllers, enabling efficient composition of policies and demonstrating successful transfer from simulation to real-world robot obstacle avoidance.

Contribution

It proposes a structured sparse kernel RL algorithm with compositional capabilities, improving sample efficiency and enabling policy transfer to physical robots.

Findings

Policies can be composed without additional training.

Composed policies retain component performance.

Successful transfer to real robot obstacle avoidance.

Abstract

We present a reinforcement learning algorithm for learning sparse non-parametric controllers in a Reproducing Kernel Hilbert Space. We improve the sample complexity of this approach by imposing a structure of the state-action function through a normalized advantage function (NAF). This representation of the policy enables efficiently composing multiple learned models without additional training samples or interaction with the environment. We demonstrate the performance of this algorithm on learning obstacle-avoidance policies in multiple simulations of a robot equipped with a laser scanner while navigating in a 2D environment. We apply the composition operation to various policy combinations and test them to show that the composed policies retain the performance of their components. We also transfer the composed policy directly to a physical platform operating in an arena with obstacles in order to demonstrate a degree of generalization.