Developing cooperative policies for multi-stage tasks

TL;DR

This paper introduces the Cooperative Soft Actor Critic (CSAC) method, enabling reinforcement learning agents to cooperatively solve multi-stage tasks by optimizing both current and future rewards, leading to improved performance.

Contribution

The paper presents CSAC, a novel multi-agent reinforcement learning approach that enhances cooperation across stages by modifying agent policies to maximize both current and next critic.

Findings

CSAC outperforms uncooperative policies by at least 20% success rate.

CSAC converges four times faster than a single agent approach.

Cooperative policies excelled in a multi-room maze domain.

Abstract

This paper proposes the Cooperative Soft Actor Critic (CSAC) method of enabling consecutive reinforcement learning agents to cooperatively solve a long time horizon multi-stage task. This method is achieved by modifying the policy of each agent to maximise both the current and next agent's critic. Cooperatively maximising each agent's critic allows each agent to take actions that are beneficial for its task as well as subsequent tasks. Using this method in a multi-room maze domain, the cooperative policies were able to outperform both uncooperative policies as well as a single agent trained across the entire domain. CSAC achieved a success rate of at least 20\% higher than the uncooperative policies, and converged on a solution at least 4 times faster than the single agent.