One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning

TL;DR

This paper presents GenZ-LTL, a novel approach enabling zero-shot generalization to complex LTL-specified tasks in reinforcement learning by decomposing tasks into subgoals and solving them sequentially, improving over prior methods.

Contribution

The paper introduces GenZ-LTL, a method that decomposes LTL tasks into subgoals using Büchi automata and solves them sequentially, enhancing zero-shot generalization capabilities.

Findings

Outperforms existing methods in zero-shot generalization to unseen LTL tasks.

Effectively decomposes complex LTL tasks into manageable subgoals.

Mitigates exponential complexity with a novel observation reduction technique.

Abstract

Generalizing to complex and temporally extended task objectives and safety constraints remains a critical challenge in reinforcement learning (RL). Linear temporal logic (LTL) offers a unified formalism to specify such requirements, yet existing methods are limited in their abilities to handle nested long-horizon tasks and safety constraints, and cannot identify situations when a subgoal is not satisfiable and an alternative should be sought. In this paper, we introduce GenZ-LTL, a method that enables zero-shot generalization to arbitrary LTL specifications. GenZ-LTL leverages the structure of B\"uchi automata to decompose an LTL task specification into sequences of reach-avoid subgoals. Contrary to the current state-of-the-art method that conditions on subgoal sequences, we show that it is more effective to achieve zero-shot generalization by solving these reach-avoid problems \textit{one subgoal at a time} through proper safe RL formulations. In addition, we introduce a novel subgoal-induced observation reduction technique that can mitigate the exponential complexity of subgoal-state combinations under realistic assumptions. Empirical results show that GenZ-LTL substantially outperforms existing methods in zero-shot generalization to unseen LTL specifications.