Improving performance in multi-objective decision-making in Bottles environments with soft maximin approaches

TL;DR

This paper introduces SFELLA, a novel soft maximin approach for multi-objective decision-making that learns faster and is more robust than existing methods, improving AI alignment in complex environments.

Contribution

The paper proposes SFELLA, a new loss-averse soft maximin method that outperforms state-of-the-art approaches in learning speed and robustness in multi-objective AI tasks.

Findings

SFELLA learns faster than thresholded methods in most tasks.

SFELLA maintains optimal performance after learning.

SFELLA shows robustness to changes in objective scale.

Abstract

Balancing multiple competing and conflicting objectives is an essential task for any artificial intelligence tasked with satisfying human values or preferences. Conflict arises both from misalignment between individuals with competing values, but also between conflicting value systems held by a single human. Starting with principle of loss-aversion, we designed a set of soft maximin function approaches to multi-objective decision-making. Bench-marking these functions in a set of previously-developed environments, we found that one new approach in particular, 'split-function exp-log loss aversion' (SFELLA), learns faster than the state of the art thresholded alignment objective method (Vamplew et al, 2021) on three of four tasks it was tested on, and achieved the same optimal performance after learning. SFELLA also showed relative robustness improvements against changes in objective scale, which may highlight an advantage dealing with distribution shifts in the environment dynamics. Due to publishing rules, further work could not be presented in the preprint, but in the final published version, we will further compare SFELLA to the multi-objective reward exponentials (MORE) approach (Rolf, 2020), demonstrating that SFELLA performs similarly to MORE in a simple previously-described foraging task, but in a modified foraging environment with a new resource that was not depleted as the agent worked, SFELLA collected more of the new resource with very little cost incurred in terms of the old resource. Overall, we found SFELLA useful for avoiding problems that sometimes occur with a thresholded approach, and more reward-responsive than MORE while retaining its conservative, loss-averse incentive structure.