Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classification

TL;DR

This paper introduces Loss Conditional Training (LCT), a method that trains models over a distribution of hyperparameters to improve performance and adaptability in imbalanced classification tasks, especially in medical imaging.

Contribution

The paper proposes LCT, a novel training approach that enhances model performance and flexibility by optimizing over hyperparameter distributions rather than fixed values.

Findings

LCT improves performance on CIFAR and medical imaging datasets.

Training with LCT is more efficient, allowing hyperparameter tuning post-training.

LCT approximates multiple models' performance, offering better adaptability.

Abstract

Although binary classification is a well-studied problem, training reliable classifiers under severe class imbalance remains a challenge. Recent techniques mitigate the ill effects of imbalance on training by modifying the loss functions or optimization methods. We observe that different hyperparameter values on these loss functions perform better at different recall values. We propose to exploit this fact by training one model over a distribution of hyperparameter values--instead of a single value--via Loss Conditional Training (LCT). Experiments show that training over a distribution of hyperparameters not only approximates the performance of several models but actually improves the overall performance of models on both CIFAR and real medical imaging applications, such as melanoma and diabetic retinopathy detection. Furthermore, training models with LCT is more efficient because some hyperparameter tuning can be conducted after training to meet individual needs without needing to retrain from scratch.