Per-Domain Generalizing Policies: On Learning Efficient and Robust Q-Value Functions (Extended Version with Technical Appendix)

TL;DR

This paper proposes learning Q-value functions for per-domain generalizing policies, which are more efficient and robust than state-value functions, by introducing regularization techniques that improve their performance across multiple domains.

Contribution

It introduces a novel approach of learning Q-value functions with regularization to enhance efficiency and robustness in domain-generalized policies, outperforming traditional state-value methods.

Findings

Q-value policies outperform state-value policies in 10 domains

Regularization improves the distinction between actions taken and not taken

Q-value policies are competitive with the planner LAMA-first

Abstract

Learning per-domain generalizing policies is a key challenge in learning for planning. Standard approaches learn state-value functions represented as graph neural networks using supervised learning on optimal plans generated by a teacher planner. In this work, we advocate for learning Q-value functions instead. Such policies are drastically cheaper to evaluate for a given state, as they need to process only the current state rather than every successor. Surprisingly, vanilla supervised learning of Q-values performs poorly as it does not learn to distinguish between the actions taken and those not taken by the teacher. We address this by using regularization terms that enforce this distinction, resulting in Q-value policies that consistently outperform state-value policies across a range of 10 domains and are competitive with the planner LAMA-first.