Deep Minimax Classifiers for Imbalanced Datasets with a Small Number of Minority Samples

TL;DR

This paper introduces a novel minimax learning algorithm for neural networks that effectively handles imbalanced datasets with few minority samples by iteratively optimizing class priors and decision boundaries, with theoretical and empirical validation.

Contribution

The paper proposes a new minimax training algorithm with a targeted loss function and prior updating strategy, improving generalization and robustness in imbalanced, small-sample scenarios.

Findings

Outperforms existing methods on imbalanced datasets.

Theoretically proven to have better generalization bounds.

Demonstrates robustness with small sample sizes.

Abstract

The concept of a minimax classifier is well-established in statistical decision theory, but its implementation via neural networks remains challenging, particularly in scenarios with imbalanced training data having a limited number of samples for minority classes. To address this issue, we propose a novel minimax learning algorithm designed to minimize the risk of worst-performing classes. Our algorithm iterates through two steps: a minimization step that trains the model based on a selected target prior, and a maximization step that updates the target prior towards the adversarial prior for the trained model. In the minimization, we introduce a targeted logit-adjustment loss function that efficiently identifies optimal decision boundaries under the target prior. Moreover, based on a new prior-dependent generalization bound that we obtained, we theoretically prove that our loss function has a better generalization capability than existing loss functions. During the maximization, we refine the target prior by shifting it towards the adversarial prior, depending on the worst-performing classes rather than on per-class risk estimates. Our maximization method is particularly robust in the regime of a small number of samples. Additionally, to adapt to overparameterized neural networks, we partition the entire training dataset into two subsets: one for model training during the minimization step and the other for updating the target prior during the maximization step. Our proposed algorithm has a provable convergence property, and empirical results indicate that our algorithm performs better than or is comparable to existing methods. All codes are publicly available at https://github.com/hansung-choi/TLA-linear-ascent.