Correcting Class Imbalance in Prior-Data Fitted Networks for Tabular Classification

TL;DR

This paper investigates methods to address class imbalance in prior-data fitted networks (PFNs) for tabular classification, highlighting thresholding and downsampling as effective strategies.

Contribution

It adapts classical class imbalance techniques to PFNs and analyzes their effectiveness, revealing thresholding and downsampling as promising solutions.

Findings

Thresholding performs exceptionally well due to PFNs' calibration.

Downsampling is comparable and reduces inference cost.

Class imbalance mitigation improves PFN performance on rare classes.

Abstract

Prior-data fitted networks (PFNs) have achieved exceptional performance on tabular classification tasks. However, like other classifiers, their performance can suffer under the effect of class imbalance, resulting in poor performance for rare classes. Several techniques exist which attempt to mitigate the deleterious effect of class imbalance on classification performance, but the in-context learning (ICL) dynamic of PFNs means that loss-based strategies are impossible, and other techniques are unproven. We have adapted several classical techniques addressing class imbalance and analyzed their performance on PFN classification. We observe that thresholding performs exceptionally well because of the calibration characteristics of PFNs, and downsampling performs comparably because of PFNs exceptional limited-data performance, with the additional benefit of reduced computation cost for inference.