Continuous Value Iteration (CVI) Reinforcement Learning and Imaginary Experience Replay (IER) for learning multi-goal, continuous action and state space controllers

TL;DR

This paper introduces a new model-free reinforcement learning algorithm that efficiently learns multi-goal controllers in continuous spaces, utilizing non-parametric value function approximation and a novel sample augmentation technique to enhance generalization and learning speed.

Contribution

It proposes a novel RL algorithm combining continuous value iteration and imaginary experience replay, improving multi-goal learning in continuous spaces with better generalization.

Findings

Faster learning in simulation and real-world robot tasks.

Effective multi-goal control in continuous action and state spaces.

Enhanced generalization through sample augmentation.

Abstract

This paper presents a novel model-free Reinforcement Learning algorithm for learning behavior in continuous action, state, and goal spaces. The algorithm approximates optimal value functions using non-parametric estimators. It is able to efficiently learn to reach multiple arbitrary goals in deterministic and nondeterministic environments. To improve generalization in the goal space, we propose a novel sample augmentation technique. Using these methods, robots learn faster and overall better controllers. We benchmark the proposed algorithms using simulation and a real-world voltage controlled robot that learns to maneuver in a non-observable Cartesian task space.