Transferring Multiple Policies to Hotstart Reinforcement Learning in an Air Compressor Management Problem

TL;DR

This paper introduces a method to transfer multiple pre-trained policies to accelerate reinforcement learning for new but similar industrial machine control tasks, demonstrated on an air compressor management problem.

Contribution

It applies Policy Intersection to transfer knowledge from several controllers, improving learning speed and performance in compressor control tasks.

Findings

Outperforms loading a single old controller.

Significantly improves long-term performance.

Reduces training time and resources.

Abstract

Many instances of similar or almost-identical industrial machines or tools are often deployed at once, or in quick succession. For instance, a particular model of air compressor may be installed at hundreds of customers. Because these tools perform distinct but highly similar tasks, it is interesting to be able to quickly produce a high-quality controller for machine $N+1$ given the controllers already produced for machines $1..N$. This is even more important when the controllers are learned through Reinforcement Learning, as training takes time, energy and other resources. In this paper, we apply Policy Intersection, a Policy Shaping method, to help a Reinforcement Learning agent learn to solve a new variant of a compressors control problem faster, by transferring knowledge from several previously learned controllers. We show that our approach outperforms loading an old controller, and significantly improves performance in the long run.