DSDF: An approach to handle stochastic agents in collaborative multi-agent reinforcement learning

TL;DR

This paper introduces DSDF, a method that adjusts for stochasticity in multi-agent reinforcement learning, improving coordination and convergence when agents have varying levels of reliability or malfunction.

Contribution

The paper proposes DSDF, a novel approach that tunes the discount factor based on agent uncertainty, enhancing coordination among stochastic and deterministic agents.

Findings

DSDF improves convergence in stochastic multi-agent settings.

The method enhances coordination reliability among agents.

Results outperform existing approaches on benchmark environments.

Abstract

Multi-Agent reinforcement learning has received lot of attention in recent years and have applications in many different areas. Existing methods involving Centralized Training and Decentralized execution, attempts to train the agents towards learning a pattern of coordinated actions to arrive at optimal joint policy. However if some agents are stochastic to varying degrees of stochasticity, the above methods often fail to converge and provides poor coordination among agents. In this paper we show how this stochasticity of agents, which could be a result of malfunction or aging of robots, can add to the uncertainty in coordination and there contribute to unsatisfactory global coordination. In this case, the deterministic agents have to understand the behavior and limitations of the stochastic agents while arriving at optimal joint policy. Our solution, DSDF which tunes the discounted factor for the agents according to uncertainty and use the values to update the utility networks of individual agents. DSDF also helps in imparting an extent of reliability in coordination thereby granting stochastic agents tasks which are immediate and of shorter trajectory with deterministic ones taking the tasks which involve longer planning. Such an method enables joint co-ordinations of agents some of which may be partially performing and thereby can reduce or delay the investment of agent/robot replacement in many circumstances. Results on benchmark environment for different scenarios shows the efficacy of the proposed approach when compared with existing approaches.