AdaCap: An Adaptive Contrastive Approach for Small-Data Neural Networks

TL;DR

AdaCap is a novel training scheme that significantly improves neural network performance on small tabular datasets by combining contrastive loss with a closed-form output mapping, acting as targeted regularization.

Contribution

The paper introduces AdaCap, a new contrastive training method that enhances neural network performance on small datasets, outperforming traditional models and predicting when it is most effective.

Findings

AdaCap improves neural network accuracy on small datasets.

It outperforms traditional models in regression tasks.

The meta-predictor accurately forecasts AdaCap's benefits.

Abstract

Neural networks struggle on small tabular datasets, where tree-based models remain dominant. We introduce Adaptive Contrastive Approach (AdaCap), a training scheme that combines a permutation-based contrastive loss with a Tikhonov-based closed-form output mapping. Across 85 real-world regression datasets and multiple architectures, AdaCap yields consistent and statistically significant improvements in the small-sample regime, particularly for residual models. A meta-predictor trained on dataset characteristics (size, skewness, noise) accurately anticipates when AdaCap is beneficial. These results show that AdaCap acts as a targeted regularization mechanism, strengthening neural networks precisely where they are most fragile. All results and code are publicly available at https://github.com/BrunoBelucci/adacap.