Post Hoc Regression Refinement via Pairwise Rankings

TL;DR

RankRefine is a versatile post hoc method that enhances regression accuracy in data-scarce scenarios by integrating pairwise rankings from experts or LLMs without retraining, demonstrated in molecular property prediction.

Contribution

Introduces RankRefine, a model-agnostic post hoc approach that refines regressions using pairwise rankings, requiring no retraining and applicable across domains.

Findings

Achieves up to 10% reduction in mean absolute error in molecular property prediction.

Uses only 20 pairwise comparisons from LLMs or experts.

Applicable in low-data regimes and diverse domains.

Abstract

Accurate prediction of continuous properties is essential to many scientific and engineering tasks. Although deep-learning regressors excel with abundant labels, their accuracy deteriorates in data-scarce regimes. We introduce RankRefine, a model-agnostic, plug-and-play post hoc method that refines regression with expert knowledge coming from pairwise rankings. Given a query item and a small reference set with known properties, RankRefine combines the base regressor's output with a rank-based estimate via inverse variance weighting, requiring no retraining. In molecular property prediction task, RankRefine achieves up to 10% relative reduction in mean absolute error using only 20 pairwise comparisons obtained through a general-purpose large language model (LLM) with no finetuning. As rankings provided by human experts or general-purpose LLMs are sufficient for improving regression across diverse domains, RankRefine offers practicality and broad applicability, especially in low-data settings.