Data-Efficiency with a Single GPU: An Exploration of Transfer Methods for Small Language Models

TL;DR

This paper investigates how various transfer learning methods like multi-task learning and instruction tuning affect small language models under 500 million parameters, revealing that MTL significantly improves performance, while instruction tuning offers modest gains.

Contribution

The study systematically isolates and evaluates the effects of different transfer methods on small language models, providing new insights contrasting with large model findings.

Findings

General purpose MTL yields 31% relative improvement.

In-domain MTL adds 37.6% relative gain.

Instruction tuning offers only 2% performance boost.

Abstract

Multi-task learning (MTL), instruction tuning, and prompting have recently been shown to improve the generalizability of large language models to new tasks. However, the benefits of such methods are less well-documented in smaller language models, with some studies finding contradictory results. In this work, we explore and isolate the effects of (i) model size, (ii) general purpose MTL, (iii) in-domain MTL, (iv) instruction tuning, and (v) few-shot fine-tuning for models with fewer than 500 million parameters. Our experiments in the zero-shot setting demonstrate that models gain 31% relative improvement, on average, from general purpose MTL, with an additional 37.6% relative gain from in-domain MTL. Contradictory to prior works on large models, we find that instruction tuning provides a modest 2% performance improvement for small models.