CAIRO: Decoupling Order from Scale in Regression

TL;DR

CAIRO introduces a two-stage regression framework that decouples order learning from scale calibration, enhancing robustness to outliers and heavy-tailed noise, and achieves state-of-the-art results on tabular benchmarks.

Contribution

It proposes a novel decoupled regression method, CAIRO, with theoretical guarantees for order recovery and calibration, improving robustness and performance.

Findings

Matches state-of-the-art on tabular benchmarks

Outperforms standard regression with heavy-tailed noise

Guarantees auto-calibration at the population level

Abstract

Standard regression methods typically optimize a single pointwise objective, such as mean squared error, which conflates the learning of ordering with the learning of scale. This coupling renders models vulnerable to outliers and heavy-tailed noise. We propose CAIRO (Calibrate After Initial Rank Ordering), a framework that decouples regression into two distinct stages. In the first stage, we learn a scoring function by minimizing a scale-invariant ranking loss; in the second, we recover the target scale via isotonic regression. We theoretically characterize a class of "Optimal-in-Rank-Order" objectives -- including variants of RankNet and Gini covariance -- and prove that they recover the ordering of the true conditional mean under mild assumptions. We further show that subsequent monotone calibration recovers the true regression function at the population level and mathematically guarantees that finite-sample predictions are strictly auto-calibrated. Empirically, CAIRO combines the representation learning of neural networks with the robustness of rank-based statistics. It matches the performance of state-of-the-art tree ensembles on tabular benchmarks and significantly outperforms standard regression objectives in regimes with heavy-tailed or heteroskedastic noise.