Lattice-based Deep Neural Networks: Regularity and Tailored Regularization

TL;DR

This paper reviews the application of lattice rules to Deep Neural Networks, demonstrating their effectiveness in improving generalization and regularization, with theoretical bounds and numerical evidence supporting their advantages over standard methods.

Contribution

It introduces a novel approach of using lattice rules as training points for DNNs, providing explicit regularity bounds and demonstrating improved performance through tailored regularization.

Findings

Lattice rules can be effectively used as training points for DNNs.

Tailored regularization based on lattice rules improves generalization.

Numerical experiments show better performance than standard regularization.

Abstract

This survey article is concerned with the application of lattice rules to Deep Neural Networks (DNNs), lattice rules being a family of quasi-Monte Carlo methods. They have demonstrated effectiveness in various contexts for high-dimensional integration and function approximation. They are extremely easy to implement thanks to their very simple formulation -- all that is required is a good integer generating vector of length matching the dimensionality of the problem. In recent years there has been a burst of research activities on the application and theory of DNNs. We review our recent article on using lattice rules as training points for DNNs with a smooth activation function, where we obtained explicit regularity bounds of the DNNs. By imposing restrictions on the network parameters to match the regularity features of the target function, we prove that DNNs with tailored lattice training points can achieve good theoretical generalization error bounds, with implied constants independent of the input dimension. We also demonstrate numerically that DNNs trained with our tailored regularization perform significantly better than with standard $\ell_2$ regularization.