ParMod: A Parallel and Modular Framework for Learning Non-Markovian Tasks

TL;DR

ParMod introduces a parallel and modular reinforcement learning framework designed to efficiently learn non-Markovian tasks specified by temporal logic, addressing challenges of long-term dependencies and reward sparsity.

Contribution

The paper presents a novel framework that modularizes non-Markovian tasks and trains agents in parallel, improving sample efficiency and performance in complex temporal logic-based tasks.

Findings

ParMod outperforms existing methods on benchmark problems.

The modular approach enhances learning efficiency for non-Markovian tasks.

Parallel training accelerates convergence and improves task performance.

Abstract

The commonly used Reinforcement Learning (RL) model, MDPs (Markov Decision Processes), has a basic premise that rewards depend on the current state and action only. However, many real-world tasks are non-Markovian, which has long-term memory and dependency. The reward sparseness problem is further amplified in non-Markovian scenarios. Hence learning a non-Markovian task (NMT) is inherently more difficult than learning a Markovian one. In this paper, we propose a novel \textbf{Par}allel and \textbf{Mod}ular RL framework, ParMod, specifically for learning NMTs specified by temporal logic. With the aid of formal techniques, the NMT is modulaized into a series of sub-tasks based on the automaton structure (equivalent to its temporal logic counterpart). On this basis, sub-tasks will be trained by a group of agents in a parallel fashion, with one agent handling one sub-task. Besides parallel training, the core of ParMod lies in: a flexible classification method for modularizing the NMT, and an effective reward shaping method for improving the sample efficiency. A comprehensive evaluation is conducted on several challenging benchmark problems with respect to various metrics. The experimental results show that ParMod achieves superior performance over other relevant studies. Our work thus provides a good synergy among RL, NMT and temporal logic.