GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning

TL;DR

GraphAllocBench introduces a versatile, graph-based benchmark for preference-conditioned multi-objective policy learning, enabling realistic, scalable evaluation of algorithms in complex resource allocation scenarios with new metrics for preference consistency.

Contribution

It presents a novel graph-based resource allocation benchmark with diverse objectives and preferences, along with new evaluation metrics, facilitating advanced research in preference-conditioned reinforcement learning.

Findings

Existing MORL approaches have limitations exposed by GraphAllocBench.

Graph Neural Networks show promise in complex, high-dimensional allocation tasks.

The benchmark's flexibility allows for comprehensive evaluation of preference-conditioned policies.

Abstract

Preference-Conditioned Policy Learning (PCPL) in Multi-Objective Reinforcement Learning (MORL) aims to approximate diverse Pareto-optimal solutions by conditioning policies on user-specified preferences over objectives. This enables a single model to flexibly adapt to arbitrary trade-offs at run-time by producing a policy on or near the Pareto front. However, existing benchmarks for PCPL are largely restricted to toy tasks and fixed environments, limiting their realism and scalability. To address this gap, we introduce GraphAllocBench, a flexible benchmark built on a novel graph-based resource allocation sandbox environment inspired by city management, which we call CityPlannerEnv. GraphAllocBench provides a rich suite of problems with diverse objective functions, varying preference conditions, and high-dimensional scalability. We also propose two new evaluation metrics -- Proportion of Non-Dominated Solutions (PNDS) and Ordering Score (OS) -- that directly capture preference consistency while complementing the widely used hypervolume metric. Through experiments with Multi-Layer Perceptrons (MLPs) and graph-aware models, we show that GraphAllocBench exposes the limitations of existing MORL approaches and paves the way for using graph-based methods such as Graph Neural Networks (GNNs) in complex, high-dimensional combinatorial allocation tasks. Beyond its predefined problem set, GraphAllocBench enables users to flexibly vary objectives, preferences, and allocation rules, establishing it as a versatile and extensible benchmark for advancing PCPL. Code: https://github.com/jzh001/GraphAllocBench