On the Effect of Instruction Tuning Loss on Generalization

TL;DR

This paper investigates how different weighting schemes for prompt and response tokens in instruction tuning loss affect model performance, proposing Weighted Instruction Tuning (WIT) to improve robustness and generalization.

Contribution

It introduces WIT, a novel loss weighting method for instruction tuning, demonstrating its effectiveness across multiple models, datasets, and benchmarks.

Findings

Optimal prompt and response weightings improve performance.

Standard instruction tuning loss often limits robustness.

WIT enhances model generalization and robustness.

Abstract

Instruction Tuning has emerged as a pivotal post-training paradigm that enables pre-trained language models to better follow user instructions. Despite its significance, little attention has been given to optimizing the loss function used. A fundamental, yet often overlooked, question is whether the conventional auto-regressive objective - where loss is computed only on response tokens, excluding prompt tokens - is truly optimal for instruction tuning. In this work, we systematically investigate the impact of differentially weighting prompt and response tokens in instruction tuning loss, and propose Weighted Instruction Tuning (WIT) as a better alternative to conventional instruction tuning. Through extensive experiments on five language models of different families and scale, three finetuning datasets of different sizes, and five diverse evaluation benchmarks, we show that the standard instruction tuning loss often yields suboptimal performance and limited robustness to input prompt variations. We find that a low-to-moderate weight for prompt tokens coupled with a moderate-to-high weight for response tokens yields the best-performing models across settings and also serve as better starting points for the subsequent preference alignment training. These findings highlight the need to reconsider instruction tuning loss and offer actionable insights for developing more robust and generalizable models. Our code is open-sourced at https://github.com/kowndinya-renduchintala/WIT.