A Note on Dimensionality Reduction in Deep Neural Networks using Empirical Interpolation Method

TL;DR

This paper introduces DNN-EIM, a method combining empirical interpolation with neural networks to reduce training data dimensionality, enabling faster training with minimal accuracy loss in data science and PDE applications.

Contribution

The paper presents a novel integration of empirical interpolation with deep neural networks, allowing parallel training and incremental class addition without retraining.

Findings

Training times are significantly reduced.

Networks require fewer than ten times the training weights.

Accuracy is maintained despite dimensionality reduction.

Abstract

Empirical interpolation method (EIM) is a well-known technique to efficiently approximate parameterized functions. This paper proposes to use EIM algorithm to efficiently reduce the dimension of the training data within supervised machine learning. This is termed as DNN-EIM. Applications in data science (e.g., MNIST) and parameterized (and time-dependent) partial differential equations (PDEs) are considered. The proposed DNNs in case of classification are trained in parallel for each class. This approach is sequential, i.e., new classes can be added without having to retrain the network. In case of PDEs, a DNN is designed corresponding to each EIM point. Again, these networks can be trained in parallel, for each EIM point. In all cases, the parallel networks require fewer than ten times the number of training weights. Significant gains are observed in terms of training times, without sacrificing accuracy.