Sensitivity of Small Language Models to Fine-tuning Data Contamination

TL;DR

This paper systematically investigates how small language models are affected by data contamination during instruction tuning, revealing significant vulnerabilities especially to syntactic transformations and an unexpected susceptibility increase in larger models.

Contribution

It provides empirical evidence of contamination vulnerabilities in small language models, highlighting asymmetric effects of syntactic and semantic transformations, and proposes evaluation protocols for robustness assessment.

Findings

Syntactic transformations cause near-complete failure across models.

Larger models are more susceptible to semantic contamination.

Alignment sometimes reduces robustness, not always improving resilience.

Abstract

Small Language Models (SLMs) are increasingly being deployed in resource-constrained environments, yet their behavioral robustness to data contamination during instruction tuning remains poorly understood. We systematically investigate the contamination sensitivity of 23 SLMs (270M to 4B parameters) across multiple model families by measuring susceptibility to syntactic and semantic transformation types during instruction tuning: syntactic transformations (character and word reversal) and semantic transformations (irrelevant and counterfactual responses), each applied at contamination levels of 25\%, 50\%, 75\%, and 100\%. Our results reveal fundamental asymmetries in vulnerability patterns: syntactic transformations cause catastrophic performance degradation, with character reversal producing near-complete failure across all models regardless of size or family, while semantic transformations demonstrate distinct threshold behaviors and greater resilience in core linguistic capabilities. Critically, we discover a ``\textit{capability curse}" where larger, more capable models become more susceptible to learning semantic corruptions, effectively following harmful instructions more readily, while our analysis of base versus instruction-tuned variants reveals that alignment provides inconsistent robustness benefits, sometimes even reducing resilience. Our work establishes three core contributions: (1) empirical evidence of SLMs' disproportionate vulnerability to syntactic pattern contamination, (2) identification of asymmetric sensitivity patterns between syntactic and semantic transformations, and (3) systematic evaluation protocols for contamination robustness assessment. These findings have immediate deployment implications, suggesting that current robustness assumptions may not hold for smaller models and highlighting the need for contamination-aware training protocols.