Scalable Subsampling Inference for Deep Neural Networks

TL;DR

This paper introduces a scalable subsampling method for deep neural networks that enhances computational efficiency and enables reliable statistical inference, including confidence and prediction intervals, without sacrificing accuracy.

Contribution

It develops a scalable subsampling approach for DNNs that improves inference efficiency and provides valid confidence and prediction intervals in finite samples.

Findings

Subagged DNN estimator is computationally efficient.

The method produces valid confidence and prediction intervals.

Improved error bounds for DNN approximation.

Abstract

Deep neural networks (DNN) has received increasing attention in machine learning applications in the last several years. Recently, a non-asymptotic error bound has been developed to measure the performance of the fully connected DNN estimator with ReLU activation functions for estimating regression models. The paper at hand gives a small improvement on the current error bound based on the latest results on the approximation ability of DNN. More importantly, however, a non-random subsampling technique--scalable subsampling--is applied to construct a `subagged' DNN estimator. Under regularity conditions, it is shown that the subagged DNN estimator is computationally efficient without sacrificing accuracy for either estimation or prediction tasks. Beyond point estimation/prediction, we propose different approaches to build confidence and prediction intervals based on the subagged DNN estimator. In addition to being asymptotically valid, the proposed confidence/prediction intervals appear to work well in finite samples. All in all, the scalable subsampling DNN estimator offers the complete package in terms of statistical inference, i.e., (a) computational efficiency; (b) point estimation/prediction accuracy; and (c) allowing for the construction of practically useful confidence and prediction intervals.