Adaptable Automation with Modular Deep Reinforcement Learning and Policy Transfer

TL;DR

This paper introduces HASAC, a modular deep RL framework that improves adaptability and transfer learning for robotic tasks, demonstrating superior performance on a new benchmark.

Contribution

The paper develops HASAC, a novel RL framework that enhances task transferability and adaptability through modularization and a hyper-actor, addressing limitations of existing deep RL methods.

Findings

HASAC outperforms state-of-the-art algorithms in reward and success rate.

HASAC demonstrates faster task completion times.

The framework shows effective transfer learning on the Meta-World benchmark.

Abstract

Recent advances in deep Reinforcement Learning (RL) have created unprecedented opportunities for intelligent automation, where a machine can autonomously learn an optimal policy for performing a given task. However, current deep RL algorithms predominantly specialize in a narrow range of tasks, are sample inefficient, and lack sufficient stability, which in turn hinder their industrial adoption. This article tackles this limitation by developing and testing a Hyper-Actor Soft Actor-Critic (HASAC) RL framework based on the notions of task modularization and transfer learning. The goal of the proposed HASAC is to enhance the adaptability of an agent to new tasks by transferring the learned policies of former tasks to the new task via a "hyper-actor". The HASAC framework is tested on a new virtual robotic manipulation benchmark, Meta-World. Numerical experiments show superior performance by HASAC over state-of-the-art deep RL algorithms in terms of reward value, success rate, and task completion time.