Efficient Multi-Task Reinforcement Learning with Cross-Task Policy Guidance

TL;DR

This paper introduces Cross-Task Policy Guidance (CTPG), a framework that uses proficient task policies to guide unmastered tasks in multi-task reinforcement learning, significantly improving learning efficiency and performance.

Contribution

The paper proposes a novel CTPG framework that explicitly leverages cross-task policy guidance and gating mechanisms to enhance multi-task reinforcement learning.

Findings

CTPG improves learning speed in manipulation and locomotion tasks.

Incorporating CTPG significantly boosts performance over existing methods.

Gating mechanisms effectively filter beneficial policies and tasks.

Abstract

Multi-task reinforcement learning endeavors to efficiently leverage shared information across various tasks, facilitating the simultaneous learning of multiple tasks. Existing approaches primarily focus on parameter sharing with carefully designed network structures or tailored optimization procedures. However, they overlook a direct and complementary way to exploit cross-task similarities: the control policies of tasks already proficient in some skills can provide explicit guidance for unmastered tasks to accelerate skills acquisition. To this end, we present a novel framework called Cross-Task Policy Guidance (CTPG), which trains a guide policy for each task to select the behavior policy interacting with the environment from all tasks' control policies, generating better training trajectories. In addition, we propose two gating mechanisms to improve the learning efficiency of CTPG: one gate filters out control policies that are not beneficial for guidance, while the other gate blocks tasks that do not necessitate guidance. CTPG is a general framework adaptable to existing parameter sharing approaches. Empirical evaluations demonstrate that incorporating CTPG with these approaches significantly enhances performance in manipulation and locomotion benchmarks.