Symmetry-Aware Transformer Training for Automated Planning

TL;DR

This paper introduces a symmetry-aware training method for transformers in automated planning, addressing their inability to handle problem symmetries and improving their performance in plan generation and heuristic prediction.

Contribution

It proposes a novel contrastive learning objective combined with architectural improvements to make transformers symmetry-aware for automated planning tasks.

Findings

Enhanced transformer performance on planning domains

Effective handling of problem symmetries

Improved plan-generation and heuristic prediction

Abstract

While transformers excel in many settings, their application in the field of automated planning is limited. Prior work like PlanGPT, a state-of-the-art decoder-only transformer, struggles with extrapolation from easy to hard planning problems. This in turn stems from problem symmetries: planning tasks can be represented with arbitrary variable names that carry no meaning beyond being identifiers. This causes a combinatorial explosion of equivalent representations that pure transformers cannot efficiently learn from. We propose a novel contrastive learning objective to make transformers symmetry-aware and thereby compensate for their lack of inductive bias. Combining this with architectural improvements, we show that transformers can be efficiently trained for either plan-generation or heuristic-prediction. Our results across multiple planning domains demonstrate that our symmetry-aware training effectively and efficiently addresses the limitations of PlanGPT.