Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing

TL;DR

This paper develops a theoretically optimal retraining framework for binary classification using approximate message passing, demonstrating how to best combine model predictions and labels to minimize error, especially under high noise.

Contribution

It introduces a principled AMP-based analysis to derive the Bayes optimal aggregator for iterative retraining, advancing understanding of optimal model self-boosting strategies.

Findings

Derives the Bayes optimal aggregator function for retraining.

Quantifies performance improvements over multiple retraining rounds.

Proposes a practical version of the optimal aggregator for high noise scenarios.

Abstract

Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance. While prior works have demonstrated the benefits of specific heuristic retraining schemes, the question of how to optimally combine the model's predictions and the provided labels remains largely open. This paper addresses this fundamental question for binary classification tasks. We develop a principled framework based on approximate message passing (AMP) to analyze iterative retraining procedures for two ground truth settings: Gaussian mixture model (GMM) and generalized linear model (GLM). Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels, which when used to retrain the same model, minimizes its prediction error. We also quantify the performance of this optimal retraining strategy over multiple rounds. We complement our theoretical results by proposing a practically usable version of the theoretically-optimal aggregator function for linear probing with the cross-entropy loss, and demonstrate its superiority over baseline methods in the high label noise regime.