Evaluating Generalization and Representation Stability in Small LMs via Prompting, Fine-Tuning and Out-of-Distribution Prompts

TL;DR

This study compares prompting and fine-tuning in small language models, analyzing their generalization, stability, and internal representations across in-distribution and out-of-distribution tasks to guide model adaptation choices.

Contribution

It provides a comprehensive empirical comparison of prompting and fine-tuning, focusing on their robustness and internal representations in small language models.

Findings

Prompting shows greater robustness under distributional shifts.

Fine-tuning leads to more specialized internal representations.

Model scale influences adaptation strategy effectiveness.

Abstract

We investigate the generalization capabilities of small language models under two popular adaptation paradigms: few-shot prompting and supervised fine-tuning. While prompting is often favored for its parameter efficiency and flexibility, it remains unclear how robust this approach is in low-resource settings and under distributional shifts. This paper presents a comparative study of prompting and fine-tuning across task formats, prompt styles, and model scales, with a focus on their behavior in both in-distribution and out-of-distribution (OOD) settings. Beyond accuracy, we analyze the internal representations learned by each approach to assess the stability and abstraction of task-specific features. Our findings highlight critical differences in how small models internalize and generalize knowledge under different adaptation strategies. This work offers practical guidance for model selection in low-data regimes and contributes empirical insight into the ongoing debate over prompting versus fine-tuning. Code for the experiments is available at the following