Deep W-Networks: Solving Multi-Objective Optimisation Problems With Deep Reinforcement Learning

TL;DR

This paper introduces Deep W-Networks, a novel approach combining deep reinforcement learning with W-learning to efficiently solve multi-objective optimization problems in large state spaces, outperforming existing methods.

Contribution

The paper extends W-learning to large state spaces using deep neural networks, enabling scalable multi-objective reinforcement learning with Pareto front discovery.

Findings

DWN outperforms baseline DQN in benchmark tasks

DWN effectively finds Pareto fronts in complex environments

The approach scales to large state spaces

Abstract

In this paper, we build on advances introduced by the Deep Q-Networks (DQN) approach to extend the multi-objective tabular Reinforcement Learning (RL) algorithm W-learning to large state spaces. W-learning algorithm can naturally solve the competition between multiple single policies in multi-objective environments. However, the tabular version does not scale well to environments with large state spaces. To address this issue, we replace underlying Q-tables with DQN, and propose an addition of W-Networks, as a replacement for tabular weights (W) representations. We evaluate the resulting Deep W-Networks (DWN) approach in two widely-accepted multi-objective RL benchmarks: deep sea treasure and multi-objective mountain car. We show that DWN solves the competition between multiple policies while outperforming the baseline in the form of a DQN solution. Additionally, we demonstrate that the proposed algorithm can find the Pareto front in both tested environments.