Deep Neural Networks Are Congestion Games: From Loss Landscape to Wardrop Equilibrium and Beyond

TL;DR

This paper explores the connection between deep neural networks and congestion games in game theory, offering new insights into their loss landscapes and proposing open problems for future research.

Contribution

It introduces a novel approach by applying congestion game theory to analyze DNNs, linking their loss surfaces to Wardrop equilibrium concepts.

Findings

DNNs can be modeled as congestion games.

Loss landscapes relate to Wardrop equilibrium.

Provides open problems for further research.

Abstract

The theoretical analysis of deep neural networks (DNN) is arguably among the most challenging research directions in machine learning (ML) right now, as it requires from scientists to lay novel statistical learning foundations to explain their behaviour in practice. While some success has been achieved recently in this endeavour, the question on whether DNNs can be analyzed using the tools from other scientific fields outside the ML community has not received the attention it may well have deserved. In this paper, we explore the interplay between DNNs and game theory (GT), and show how one can benefit from the classic readily available results from the latter when analyzing the former. In particular, we consider the widely studied class of congestion games, and illustrate their intrinsic relatedness to both linear and non-linear DNNs and to the properties of their loss surface. Beyond retrieving the state-of-the-art results from the literature, we argue that our work provides a very promising novel tool for analyzing the DNNs and support this claim by proposing concrete open problems that can advance significantly our understanding of DNNs when solved.