On the regularized risk of distributionally robust learning over deep neural networks

TL;DR

This paper investigates the connection between distributionally robust learning and regularization in deep neural networks, deriving new approximations and scalable algorithms for robust training using optimal transport and control theory.

Contribution

It introduces novel regularization approximations for distributionally robust problems and develops scalable algorithms based on optimal control principles for training robust neural networks.

Findings

Derives first and second order regularized risk approximations.

Identifies mean-field optimal control structure in ResNet models.

Proposes scalable algorithms motivated by Pontryagin maximum principles.

Abstract

In this paper we explore the relation between distributionally robust learning and different forms of regularization to enforce robustness of deep neural networks. In particular, starting from a concrete min-max distributionally robust problem, and using tools from optimal transport theory, we derive first order and second order approximations to the distributionally robust problem in terms of appropriate regularized risk minimization problems. In the context of deep ResNet models, we identify the structure of the resulting regularization problems as mean-field optimal control problems where the number and dimension of state variables is within a dimension-free factor of the dimension of the original unrobust problem. Using the Pontryagin maximum principles associated to these problems we motivate a family of scalable algorithms for the training of robust neural networks. Our analysis recovers some results and algorithms known in the literature (in settings explained throughout the paper) and provides many other theoretical and algorithmic insights that to our knowledge are novel. In our analysis we employ tools that we deem useful for a future analysis of more general adversarial learning problems.