PARIS: Pruning Algorithm via the Representer theorem for Imbalanced Scenarios

TL;DR

PARIS introduces a novel neural network pruning method based on the representer theorem, effectively reducing training data size and improving rare-event regression performance without retraining.

Contribution

It proposes a closed-form residual for dataset pruning in neural networks, enabling efficient, principled removal of uninformative samples in imbalanced regression tasks.

Findings

Reduces training set size by up to 75%

Outperforms re-weighting and oversampling methods

Improves RMSE on real-world space weather data

Abstract

The challenge of \textbf{imbalanced regression} arises when standard Empirical Risk Minimization (ERM) biases models toward high-frequency regions of the data distribution, causing severe degradation on rare but high-impact ``tail'' events. Existing strategies uch as loss re-weighting or synthetic over-sampling often introduce noise, distort the underlying distribution, or add substantial algorithmic complexity. We introduce \textbf{PARIS} (Pruning Algorithm via the Representer theorem for Imbalanced Scenarios), a principled framework that mitigates imbalance by \emph{optimizing the training set itself}. PARIS leverages the representer theorem for neural networks to compute a \textbf{closed-form representer deletion residual}, which quantifies the exact change in validation loss caused by removing a single training point \emph{without retraining}. Combined with an efficient Cholesky rank-one downdating scheme, PARIS performs fast, iterative pruning that eliminates uninformative or performance-degrading samples. We use a real-world space weather example, where PARIS reduces the training set by up to 75\% while preserving or improving overall RMSE, outperforming re-weighting, synthetic oversampling, and boosting baselines. Our results demonstrate that representer-guided dataset pruning is a powerful, interpretable, and computationally efficient approach to rare-event regression.