Provable Data Subset Selection For Efficient Neural Network Training

TL;DR

This paper introduces the first coreset construction algorithm for RBF neural networks, enabling efficient data subset selection for training deep neural networks with provable approximation guarantees.

Contribution

The paper presents a novel algorithm to construct coresets for RBFNNs, allowing for provable data reduction that preserves the network's loss and gradient approximations.

Findings

Coresets effectively approximate RBFNN loss functions.

Empirical results show accurate function approximation.

Subset selection improves training efficiency.

Abstract

Radial basis function neural networks (\emph{RBFNN}) are {well-known} for their capability to approximate any continuous function on a closed bounded set with arbitrary precision given enough hidden neurons. In this paper, we introduce the first algorithm to construct coresets for \emph{RBFNNs}, i.e., small weighted subsets that approximate the loss of the input data on any radial basis function network and thus approximate any function defined by an \emph{RBFNN} on the larger input data. In particular, we construct coresets for radial basis and Laplacian loss functions. We then use our coresets to obtain a provable data subset selection algorithm for training deep neural networks. Since our coresets approximate every function, they also approximate the gradient of each weight in a neural network, which is a particular function on the input. We then perform empirical evaluations on function approximation and dataset subset selection on popular network architectures and data sets, demonstrating the efficacy and accuracy of our coreset construction.