The Hybrid Bootstrap: A Drop-in Replacement for Dropout

TL;DR

The paper introduces the hybrid bootstrap, a regularization method that resamples features from other training points, showing it outperforms dropout across various models and tasks, with simplified hyperparameter tuning.

Contribution

It presents the hybrid bootstrap as a novel regularization technique, along with sampling methods for hyperparameter selection and application to convolutional neural networks and non-image data.

Findings

Hybrid bootstrap outperforms dropout in predictive accuracy.

Sampling techniques simplify hyperparameter tuning.

Effective on tree-based models for non-image tasks.

Abstract

Regularization is an important component of predictive model building. The hybrid bootstrap is a regularization technique that functions similarly to dropout except that features are resampled from other training points rather than replaced with zeros. We show that the hybrid bootstrap offers superior performance to dropout. We also present a sampling based technique to simplify hyperparameter choice. Next, we provide an alternative sampling technique for convolutional neural networks. Finally, we demonstrate the efficacy of the hybrid bootstrap on non-image tasks using tree-based models.