Rethinking Safe Control in the Presence of Self-Seeking Humans

TL;DR

This paper introduces a game-theoretic approach to safe control that accounts for self-seeking human behaviors, improving safety and performance in human-machine interactions by modeling strategic human responses.

Contribution

It models human strategic behavior within safe control systems using game theory, addressing a gap in existing methods that assume worst-case human uncertainties.

Findings

Better safety-performance trade-offs achieved

Outperforms deterministic worst-case methods

Highlights need to rethink safe control frameworks

Abstract

Safe control methods are often intended to behave safely even in worst-case human uncertainties. However, humans may exploit such safety-first systems, which results in greater risk for everyone. Despite their significance, no prior work has investigated and accounted for such factors in safe control. In this paper, we leverage an interaction-based payoff structure from game theory to model humans' short-sighted, self-seeking behaviors and how humans change their strategies toward machines based on prior experience. We integrate such strategic human behaviors into a safe control architecture. As a result, our approach achieves better safety and performance trade-offs when compared to both deterministic worst-case safe control techniques and equilibrium-based stochastic methods. Our findings suggest an urgent need to fundamentally rethink the safe control framework used in human-technology interaction in pursuit of greater safety for all.