Assessing Robustness to Spurious Correlations in Post-Training Language Models

TL;DR

This paper evaluates how different post-training methods for language models handle spurious correlations, revealing that robustness varies by task type and correlation nature, with no single method being universally best.

Contribution

It systematically compares three post-training algorithms across diverse tasks and spuriousness conditions, providing insights into their robustness and limitations.

Findings

Preference-based methods show robustness in mathematical reasoning.

Supervised Fine-Tuning performs better on complex, context-rich tasks.

Model performance degrades with increased spurious correlations, but varies by method.

Abstract

Supervised and preference-based fine-tuning techniques have become popular for aligning large language models (LLMs) with user intent and correctness criteria. However, real-world training data often exhibits spurious correlations -- arising from biases, dataset artifacts, or other "shortcut" features -- that can compromise a model's performance or generalization. In this paper, we systematically evaluate three post-training algorithms -- Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and KTO (Kahneman-Tversky Optimization) -- across a diverse set of synthetic tasks and spuriousness conditions. Our tasks span mathematical reasoning, constrained instruction-following, and document-grounded question answering. We vary the degree of spurious correlation (10% vs. 90%) and investigate two forms of artifacts: "Feature Ambiguity" and "Distributional Narrowness." Our results show that the models often but not always degrade under higher spuriousness. The preference-based methods (DPO/KTO) can demonstrate relative robustness in mathematical reasoning tasks. By contrast, SFT maintains stronger performance in complex, context-intensive tasks. These findings highlight that no single post-training strategy universally outperforms in all scenarios; the best choice depends on the type of target task and the nature of spurious correlations.