Fair Resource Allocation in Weakly Coupled Markov Decision Processes

TL;DR

This paper addresses fair resource allocation in weakly coupled Markov decision processes using a generalized Gini fairness measure, proposing new optimization and learning methods validated through experiments.

Contribution

It introduces a novel reduction to permutation invariant policies in homogeneous cases and develops a deep reinforcement learning approach for general settings.

Findings

Permutation invariant policies optimize fairness in homogeneous cases.

Deep RL approach effectively learns fair policies in complex scenarios.

Experimental results confirm the method's effectiveness in achieving fairness.

Abstract

We consider fair resource allocation in sequential decision-making environments modeled as weakly coupled Markov decision processes, where resource constraints couple the action spaces of $N$ sub-Markov decision processes (sub-MDPs) that would otherwise operate independently. We adopt a fairness definition using the generalized Gini function instead of the traditional utilitarian (total-sum) objective. After introducing a general but computationally prohibitive solution scheme based on linear programming, we focus on the homogeneous case where all sub-MDPs are identical. For this case, we show for the first time that the problem reduces to optimizing the utilitarian objective over the class of "permutation invariant" policies. This result is particularly useful as we can exploit Whittle index policies in the restless bandits setting while, for the more general setting, we introduce a count-proportion-based deep reinforcement learning approach. Finally, we validate our theoretical findings with comprehensive experiments, confirming the effectiveness of our proposed method in achieving fairness.