Information FOMO: The unhealthy fear of missing out on information. A method for removing misleading data for healthier models

TL;DR

This paper introduces a Bayesian sequential data selection method that filters out misleading information, enhancing model accuracy and stability without traditional data partitioning, applicable to Gaussian processes and neural networks.

Contribution

The paper presents a novel Bayesian data selection approach that dynamically couples data with the model, improving error convergence and eliminating the need for separate validation sets.

Findings

Reduces sample-wise error and prevents performance degradation.

Eliminates double descent phenomena in surrogate models.

Applicable to both Gaussian process and neural network models.

Abstract

Misleading or unnecessary data can have out-sized impacts on the health or accuracy of Machine Learning (ML) models. We present a Bayesian sequential selection method, akin to Bayesian experimental design, that identifies critically important information within a dataset, while ignoring data that is either misleading or brings unnecessary complexity to the surrogate model of choice. Our method improves sample-wise error convergence and eliminates instances where more data leads to worse performance and instabilities of the surrogate model, often termed sample-wise ``double descent''. We find these instabilities are a result of the complexity of the underlying map and linked to extreme events and heavy tails. Our approach has two key features. First, the selection algorithm dynamically couples the chosen model and data. Data is chosen based on its merits towards improving the selected model, rather than being compared strictly against other data. Second, a natural convergence of the method removes the need for dividing the data into training, testing, and validation sets. Instead, the selection metric inherently assesses testing and validation error through global statistics of the model. This ensures that key information is never wasted in testing or validation. The method is applied using both Gaussian process regression and deep neural network surrogate models.