Logic-based Task Representation and Reward Shaping in Multiagent Reinforcement Learning

TL;DR

This paper introduces a logic-based framework for multi-agent reinforcement learning that uses LTL task specifications, automaton conversion, and reward shaping to accelerate learning and improve convergence in complex environments.

Contribution

It presents a novel reward shaping method for multi-agent systems that reduces sample complexity and integrates options to handle large state and action spaces effectively.

Findings

Reward shaping significantly reduces convergence time.

Options become more relevant as system complexity increases.

The approach enables correct-by-design controllers without modeling transition dynamics.

Abstract

This paper presents an approach for accelerated learning of optimal plans for a given task represented using Linear Temporal Logic (LTL) in multi-agent systems. Given a set of options (temporally abstract actions) available to each agent, we convert the task specification into the corresponding Buchi Automaton and proceed with a model-free approach which collects transition samples and constructs a product Semi Markov Decision Process (SMDP) on-the-fly. Value-based Reinforcement Learning algorithms can then be used to synthesize a correct-by-design controller without learning the underlying transition model of the multi-agent system. The exponential sample complexity due to multiple agents is dealt with using a novel reward shaping approach. We test the proposed algorithm in a deterministic gridworld simulation for different tasks and find that the reward shaping results in significant reduction in convergence times. We also infer that using options becomes increasing more relevant as the state and action space increases in multi-agent systems.