Variance Reduced Policy Gradient Method for Multi-Objective Reinforcement Learning

TL;DR

This paper introduces a variance-reduction technique for policy gradient methods in multi-objective reinforcement learning, significantly improving sample efficiency without restrictive assumptions.

Contribution

It proposes a novel variance-reduction approach for policy gradients in MORL that enhances sample efficiency while maintaining scalability to large state-action spaces.

Findings

Reduced sample complexity in MORL policy gradients

Maintained scalability to large state-action spaces

Improved efficiency without restrictive assumptions

Abstract

Multi-Objective Reinforcement Learning (MORL) is a generalization of traditional Reinforcement Learning (RL) that aims to optimize multiple, often conflicting objectives simultaneously rather than focusing on a single reward. This approach is crucial in complex decision-making scenarios where agents must balance trade-offs between various goals, such as maximizing performance while minimizing costs. We consider the problem of MORL where the objectives are combined using a non-linear scalarization function. Just like in standard RL, policy gradient methods (PGMs) are amongst the most effective for handling large and continuous state-action spaces in MORL. However, existing PGMs for MORL suffer from high sample inefficiency, requiring large amounts of data to be effective. Previous attempts to solve this problem rely on overly strict assumptions, losing PGMs' benefits in scalability to large state-action spaces. In this work, we address the issue of sample efficiency by implementing variance-reduction techniques to reduce the sample complexity of policy gradients while maintaining general assumptions.