Approximating Discontinuous Nash Equilibrial Values of Two-Player General-Sum Differential Games

TL;DR

This paper explores hybrid and value-hardening methods to improve the learning of discontinuous Nash equilibrium solutions in two-player differential games, enhancing safety and generalization in high-dimensional robotic simulations.

Contribution

It introduces a hybrid approach combining supervised and self-supervised learning, and a value-hardening technique, to better approximate discontinuous PDE solutions in differential games.

Findings

Hybrid method outperforms others in safety with low computational cost.

Value hardening struggles to generalize in higher dimensions without supervision.

Continuous differentiability of neural activation functions is crucial for PDE learning.

Abstract

Finding Nash equilibrial policies for two-player differential games requires solving Hamilton-Jacobi-Isaacs (HJI) PDEs. Self-supervised learning has been used to approximate solutions of such PDEs while circumventing the curse of dimensionality. However, this method fails to learn discontinuous PDE solutions due to its sampling nature, leading to poor safety performance of the resulting controllers in robotics applications when player rewards are discontinuous. This paper investigates two potential solutions to this problem: a hybrid method that leverages both supervised Nash equilibria and the HJI PDE, and a value-hardening method where a sequence of HJIs are solved with a gradually hardening reward. We compare these solutions using the resulting generalization and safety performance in two vehicle interaction simulation studies with 5D and 9D state spaces, respectively. Results show that with informative supervision (e.g., collision and near-collision demonstrations) and the low cost of self-supervised learning, the hybrid method achieves better safety performance than the supervised, self-supervised, and value hardening approaches on equal computational budget. Value hardening fails to generalize in the higher-dimensional case without informative supervision. Lastly, we show that the neural activation function needs to be continuously differentiable for learning PDEs and its choice can be case dependent.