Sequential Decision Making in Stochastic Games with Incomplete Preferences over Temporal Objectives

TL;DR

This paper develops a framework for synthesizing strategies in stochastic games with incomplete preferences over temporal goals, ensuring stable, preference-aligned outcomes even in adversarial settings.

Contribution

It introduces the concept of non-dominated almost-sure winning strategies for games with incomplete preferences and demonstrates their properties as Nash equilibria.

Findings

Strategies based on the new concept are Nash equilibria in risk-averse settings.

The framework effectively handles incomplete and nuanced preferences over temporal goals.

Application to drone delivery illustrates practical utility in adversarial environments.

Abstract

Ensuring that AI systems make strategic decisions aligned with the specified preferences in adversarial sequential interactions is a critical challenge for developing trustworthy AI systems, especially when the environment is stochastic and players' incomplete preferences leave some outcomes unranked. We study the problem of synthesizing preference-satisfying strategies in two-player stochastic games on graphs where players have opposite (possibly incomplete) preferences over a set of temporal goals. We represent these goals using linear temporal logic over finite traces (LTLf), which enables modeling the nuances of human preferences where temporal goals need not be mutually exclusive and comparison between some goals may be unspecified. We introduce a solution concept of non-dominated almost-sure winning, which guarantees to achieve a most preferred outcome aligned with specified preferences while maintaining robustness against the adversarial behaviors of the opponent. Our results show that strategy profiles based on this concept are Nash equilibria in the game where players are risk-averse, thus providing a practical framework for evaluating and ensuring stable, preference-aligned outcomes in the game. Using a drone delivery example, we demonstrate that our contributions offer valuable insights not only for synthesizing rational behavior under incomplete preferences but also for designing games that motivate the desired behavior from the players in adversarial conditions.